> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Connecting BigQuery to Elementum

> Complete guide for setting up secure CloudLink connection between Google BigQuery and Elementum

Get your CloudLink connection up and running with Google BigQuery. This guide covers security architecture, service account setup, IAM permissions, and configuration steps.

## Secure Direct Access

Elementum provides secure, in-place data access to your BigQuery datasets:

<CardGroup cols={2}>
  <Card title="Authentication" icon="key">
    Service Account credentials permit Elementum access to defined datasets and tables. **You retain full control over the data and can terminate access at any time.**
  </Card>

  <Card title="In-Place Access" icon="database">
    Data stays in your BigQuery project. No data is copied or moved to external systems.
  </Card>
</CardGroup>

### How It Works

<Steps>
  <Step title="Google Cloud Side">
    A specific dataset and defined set of tables with IAM permissions (e.g., read/write, read-only) are configured to enable business processes.
  </Step>

  <Step title="Elementum Side">
    Further access to view the data within Elementum is restricted at the user level through access policies.
  </Step>
</Steps>

## Direct Connect Architecture

This connection allows for read/write access and the ability to execute workflows on your data:

<Tabs>
  <Tab title="Your Organization">
    **Your Google Cloud Project:**

    * Contains your data (datasets, tables, views)
    * Grants access to Elementum Service Account with appropriate IAM roles
    * Network policies can restrict access to known Elementum Platform IP addresses
  </Tab>

  <Tab title="Elementum Platform">
    **Elementum Access:**

    * Service Account provides in-place access to Elementum
    * No data copying or external storage
    * All queries execute in your BigQuery project
  </Tab>
</Tabs>

## Security Architecture

<CardGroup cols={2}>
  <Card title="Data Encryption" icon="shield-halved">
    **At Rest:**

    * Service Account JSON encrypted using industry-standard algorithms
    * Credentials encrypted and never returned outside internal system

    **In Transit:**

    * All traffic encrypted using TLS
    * Secure connection via BigQuery API
  </Card>

  <Card title="Access Control" icon="lock">
    **Network Security:**

    * VPC Service Controls support
    * Authorized networks configuration

    **Authentication:**

    * Service Account with JSON key
    * IAM-based role permissions
  </Card>
</CardGroup>

## Whitelist Elementum IP Addresses (Optional)

If you use VPC Service Controls or authorized networks, configure them to allow connections from these IP addresses:

<Tabs>
  <Tab title="US Region">
    ```
    44.210.166.136
    44.209.114.114
    52.72.254.246
    ```
  </Tab>

  <Tab title="Europe Region">
    ```
    18.185.13.42
    63.182.157.140
    3.65.106.188
    ```
  </Tab>
</Tabs>

<Callout type="info">
  **Note:** BigQuery uses Google's global network and doesn't require IP whitelisting by default. IP restrictions are only needed if you've configured VPC Service Controls or other network-level restrictions.
</Callout>

## Setting up Elementum Access in Google Cloud

### Prerequisites

Before starting the setup:

1. Ensure you have **Project Owner** or **IAM Admin** access in Google Cloud
2. Have the BigQuery API enabled in your project
3. Identify which datasets and tables need Elementum access
4. Plan your access level (read-only vs read/write)

### Setup Steps Overview

<Steps>
  <Step title="Create Service Account">
    Create a dedicated service account for Elementum access.
  </Step>

  <Step title="Create Platform Dataset">
    Create an empty dataset for Elementum platform operations. This is NOT where your data lives.
  </Step>

  <Step title="Grant IAM Roles">
    Assign appropriate BigQuery roles to the service account.
  </Step>

  <Step title="Generate JSON Key">
    Download the service account key file for authentication.
  </Step>
</Steps>

## Run These Steps in Google Cloud Console

<Steps>
  <Step title="Create Service Account">
    Navigate to **IAM & Admin > Service Accounts** in Google Cloud Console:

    1. Click **Create Service Account**
    2. **Service account name:** `elementum-cloudlink`
    3. **Service account ID:** `elementum-cloudlink` (auto-generated)
    4. **Description:** `Service account for Elementum CloudLink integration`
    5. Click **Create and Continue**

    <Callout type="info">
      You'll grant roles in the next steps. Click **Done** to create the service account first.
    </Callout>
  </Step>

  <Step title="Create Platform Dataset for Elementum">
    In the BigQuery Console:

    1. Select your project
    2. Click **Create Dataset**
    3. **Dataset ID:** `elementum_platform`
    4. **Data location:** Same region as your data (e.g., `US`, `EU`)
    5. Click **Create Dataset**

    <Warning>
      **Critical:** The `elementum_platform` dataset is for Elementum's internal operations. **Do NOT put your data tables here.** Your data tables should remain in your existing datasets.
    </Warning>
  </Step>

  <Step title="Grant Service Account Access to Platform Dataset">
    Grant the service account full access to the platform dataset:

    1. Click on the `elementum_platform` dataset
    2. Click **Sharing > Permissions**
    3. Click **Add Principal**
    4. **Principal:** `elementum-cloudlink@YOUR_PROJECT.iam.gserviceaccount.com`
    5. **Role:** `BigQuery Data Owner`
    6. Click **Save**

    Or via gcloud CLI:

    ```bash theme={null}
    gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
      --member="serviceAccount:elementum-cloudlink@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
      --role="roles/bigquery.dataOwner" \
      --condition="expression=resource.name.startsWith('projects/YOUR_PROJECT_ID/datasets/elementum_platform'),title=elementum-platform-access"
    ```
  </Step>

  <Step title="Grant BigQuery Job User Role">
    The service account needs to run queries:

    1. Go to **IAM & Admin > IAM**
    2. Find the `elementum-cloudlink` service account
    3. Click **Edit** (pencil icon)
    4. Click **Add Another Role**
    5. Select **BigQuery Job User**
    6. Click **Save**

    Or via gcloud CLI:

    ```bash theme={null}
    gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
      --member="serviceAccount:elementum-cloudlink@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
      --role="roles/bigquery.jobUser"
    ```
  </Step>

  <Step title="Grant Access to Your Data Datasets">
    For each dataset containing data you want to access:

    <Tabs>
      <Tab title="Read/Write Access">
        1. Navigate to your data dataset in BigQuery
        2. Click **Sharing > Permissions**
        3. Click **Add Principal**
        4. **Principal:** `elementum-cloudlink@YOUR_PROJECT.iam.gserviceaccount.com`
        5. **Role:** `BigQuery Data Editor`
        6. Click **Save**

        ```bash theme={null}
        # Via gcloud CLI
        bq add-iam-policy-binding \
          --member="serviceAccount:elementum-cloudlink@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
          --role="roles/bigquery.dataEditor" \
          YOUR_PROJECT_ID:your_dataset
        ```
      </Tab>

      <Tab title="Read-Only Access">
        1. Navigate to your data dataset in BigQuery
        2. Click **Sharing > Permissions**
        3. Click **Add Principal**
        4. **Principal:** `elementum-cloudlink@YOUR_PROJECT.iam.gserviceaccount.com`
        5. **Role:** `BigQuery Data Viewer`
        6. Click **Save**

        ```bash theme={null}
        # Via gcloud CLI
        bq add-iam-policy-binding \
          --member="serviceAccount:elementum-cloudlink@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
          --role="roles/bigquery.dataViewer" \
          YOUR_PROJECT_ID:your_dataset
        ```
      </Tab>

      <Tab title="Table-Level Access">
        For fine-grained control, grant access to specific tables:

        ```bash theme={null}
        # Grant access to specific table
        bq add-iam-policy-binding \
          --member="serviceAccount:elementum-cloudlink@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
          --role="roles/bigquery.dataViewer" \
          YOUR_PROJECT_ID:your_dataset.your_table
        ```
      </Tab>
    </Tabs>

    **IAM Roles Reference:**

    | Role                     | Permissions                          |
    | ------------------------ | ------------------------------------ |
    | **BigQuery Data Viewer** | Read tables, views, and metadata     |
    | **BigQuery Data Editor** | Read + Insert, update, delete data   |
    | **BigQuery Data Owner**  | Full control including delete tables |
    | **BigQuery Job User**    | Required to run queries              |
  </Step>

  <Step title="Generate Service Account JSON Key">
    Download the authentication credentials:

    1. Go to **IAM & Admin > Service Accounts**
    2. Click on `elementum-cloudlink` service account
    3. Go to **Keys** tab
    4. Click **Add Key > Create New Key**
    5. Select **JSON** format
    6. Click **Create**
    7. **Save the downloaded JSON file securely**

    <Warning>
      **Security:** Store this JSON key securely. It provides full access to the permissions you've granted. Never commit it to version control or share it publicly.
    </Warning>
  </Step>
</Steps>

## Configuring CloudLink in Elementum

After completing the Google Cloud setup, configure the connection in Elementum:

<Steps>
  <Step title="Navigate to CloudLink Settings">
    Go to **Settings > Cloud Links > Add Connection** and select **BigQuery**.
  </Step>

  <Step title="Enter Connection Details">
    Fill in the connection form:

    | Field                    | Value                 | Description                                            |
    | ------------------------ | --------------------- | ------------------------------------------------------ |
    | **Name**                 | `Production BigQuery` | Descriptive name for your connection                   |
    | **Project ID**           | `your-gcp-project-id` | Your Google Cloud project ID                           |
    | **Dataset ID**           | `elementum_platform`  | **The empty platform dataset (NOT your data dataset)** |
    | **Location**             | `US` or `EU`          | Dataset region (must match platform dataset location)  |
    | **Service Account JSON** | `{...}`               | Paste entire contents of the downloaded JSON key file  |
    | **Read Only**            | Toggle                | Enable for read-only access                            |

    <Warning>
      **Critical - Dataset ID Field:** Enter the empty platform dataset you created (e.g., `elementum_platform`), NOT your data dataset. If you enter your data dataset here, it will be hidden from workflow building and you won't be able to access your data.
    </Warning>
  </Step>

  <Step title="Configure Resource Scheduler">
    Set how frequently Elementum syncs with BigQuery:

    | Setting       | Default | Description                        |
    | ------------- | ------- | ---------------------------------- |
    | **Interval**  | 24      | Number of time units between syncs |
    | **Time Unit** | Hour    | Minutes, Hours, or Days            |

    <Callout type="info">
      **Cost consideration:** More frequent syncs consume more BigQuery query slots. For large datasets, consider longer intervals unless real-time data is required.
    </Callout>
  </Step>

  <Step title="Test and Save">
    Click **Save** to test the connection and save if successful:

    * Validates Service Account JSON format
    * Tests authentication with BigQuery API
    * Verifies dataset access

    <Callout type="success">
      If saved successfully, your connection is properly configured.
    </Callout>
  </Step>

  <Step title="Configure Data Access">
    After saving the connection:

    1. **Select Datasets:** Choose which datasets to expose in Elementum
    2. **Select Tables:** Pick specific tables from each dataset
    3. **Configure Field Mapping:** Map columns to Elementum field types
    4. **Set Primary Key:** Identify the unique identifier column for each table
  </Step>
</Steps>

## Verification and Testing

After completing the setup, verify everything is working correctly:

<Steps>
  <Step title="Test Service Account Access">
    In Google Cloud Console, verify the service account has the correct permissions:

    1. Go to **IAM & Admin > IAM**
    2. Find `elementum-cloudlink` service account
    3. Verify it has:
       * `BigQuery Job User` at project level
       * `BigQuery Data Owner` on `elementum_platform` dataset
       * `BigQuery Data Viewer` or `BigQuery Data Editor` on your data datasets
  </Step>

  <Step title="Test Query Access">
    You can test the service account by impersonating it:

    ```bash theme={null}
    # Using gcloud CLI
    gcloud auth activate-service-account \
      --key-file=path/to/your-key-file.json

    # Test query
    bq query --use_legacy_sql=false \
      'SELECT COUNT(*) FROM `your_project.your_dataset.your_table`'
    ```
  </Step>

  <Step title="Test in Elementum">
    1. Verify the connection shows as **Connected** in CloudLink settings
    2. Browse to the integrated table in Elementum
    3. Verify data loads correctly
    4. Test creating/updating a record (if write access was granted)
  </Step>
</Steps>

## Troubleshooting

<Accordion title="Connection Issues">
  **Cannot Connect from Elementum:**

  * Verify the Service Account JSON is complete and correctly formatted
  * Confirm the project ID matches your GCP project
  * Check that BigQuery API is enabled in the project
  * Verify VPC Service Controls aren't blocking access (if applicable)

  **"Permission denied" Error:**

  * Ensure `BigQuery Job User` role is granted at project level
  * Verify the service account email is correct in IAM policies
</Accordion>

<Accordion title="Cannot See Data Tables">
  **Tables not visible in Elementum:**

  * **Most common cause:** You entered your data dataset in the Dataset ID field instead of the platform dataset
  * Verify IAM roles are granted on the correct datasets
  * Check that the service account has `BigQuery Data Viewer` or `BigQuery Data Editor` on your data datasets
  * Confirm tables exist in the specified datasets
</Accordion>

<Accordion title="Permission Errors">
  **"Access Denied: Table not found" Errors:**

  * Verify dataset-level permissions are granted
  * Check that the service account has access to the specific table
  * Ensure the Location in Elementum matches the dataset location

  **"User does not have permission to run query" Error:**

  * Grant `BigQuery Job User` role at the project level
  * This role is required to execute any queries
</Accordion>

<Accordion title="Performance Issues">
  **Slow Query Performance:**

  * Consider using partitioned tables for large datasets
  * Review and optimize query patterns
  * Check BigQuery slot availability
  * Consider reserved slots for consistent performance

  **High Costs:**

  * Increase sync interval to reduce query frequency
  * Use table partitioning and clustering
  * Consider using BigQuery BI Engine for frequently accessed data
</Accordion>

## Security Best Practices

<CardGroup cols={2}>
  <Card title="Principle of Least Privilege" icon="shield">
    * Grant only necessary IAM roles to the service account
    * Use `BigQuery Data Viewer` when write access isn't required
    * Grant access at dataset level, not project level
    * Regularly audit IAM policies
  </Card>

  <Card title="Network Security" icon="network-wired">
    * Consider VPC Service Controls for sensitive data
    * Use authorized networks if applicable
    * Monitor BigQuery audit logs
    * Set up alerts for unusual access patterns
  </Card>

  <Card title="Key Management" icon="key">
    * Rotate service account keys periodically (recommended: every 90 days)
    * Store keys securely (use Secret Manager in production)
    * Delete unused keys
    * Consider workload identity federation for enhanced security
  </Card>

  <Card title="Monitoring" icon="chart-line">
    * Enable BigQuery audit logs
    * Monitor query costs and usage
    * Set up billing alerts
    * Review access patterns via Cloud Logging
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Configure Apps" icon="grid" href="/getting-started/fundamentals/core-concepts#processes--apps">
    Set up your first app in Elementum using your connected data
  </Card>

  <Card title="Create Automations" icon="bolt" href="../workflows/automation-system">
    Build workflows that act on your BigQuery data
  </Card>

  <Card title="Setup AI Features" icon="sparkles" href="../ai-agents/ai-overview">
    Enable AI-powered search, automations, and insights
  </Card>

  <Card title="Data Best Practices" icon="lightbulb" href="/data/data-best-practices">
    Optimize your data models for Elementum
  </Card>
</CardGroup>

## Additional Resources

<CardGroup cols={2}>
  <Card title="CloudLink Overview" icon="cloud" href="/administration/setup-cloudlink">
    Learn more about CloudLink architecture
  </Card>

  <Card title="BigQuery Documentation" icon="book" href="https://cloud.google.com/bigquery/docs">
    Official Google BigQuery documentation
  </Card>

  <Card title="IAM Best Practices" icon="shield" href="https://cloud.google.com/iam/docs/using-iam-securely">
    Google Cloud IAM security guide
  </Card>

  <Card title="Get Support" icon="life-ring" href="../support/resources">
    Contact our team for setup assistance
  </Card>
</CardGroup>

***

*This guide reflects the latest BigQuery and Elementum best practices. For additional assistance, contact [support@elementum.io](mailto:support@elementum.io).*