> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Connecting Databricks to Elementum

> Complete guide for setting up secure CloudLink connection between Databricks Lakebase Postgres and Elementum

Get your CloudLink connection up and running with Databricks Lakebase Postgres. This guide covers setting up a native PostgreSQL role and connecting via the managed PostgreSQL database.

## Secure Direct Access

Elementum provides secure, in-place data access to your Databricks Lakebase Postgres database:

<CardGroup cols={2}>
  <Card title="Authentication" icon="key">
    Native PostgreSQL role with password authenticates Elementum. **You retain full control over the data and can terminate access at any time.**
  </Card>

  <Card title="In-Place Access" icon="database">
    Data stays in your Databricks instance. No data is copied or moved to external systems.
  </Card>
</CardGroup>

### How It Works

<Steps>
  <Step title="Databricks Side">
    A Lakebase Postgres instance provides a managed PostgreSQL database. Service principal permissions control access to the database.
  </Step>

  <Step title="Elementum Side">
    Elementum connects via standard PostgreSQL protocol (port 5432) using native Postgres credentials. Further access restrictions are applied through Elementum access policies.
  </Step>
</Steps>

## What is Lakebase Postgres?

Lakebase Postgres is a fully managed, cloud-native PostgreSQL database within Databricks. Key features:

<CardGroup cols={2}>
  <Card title="Managed PostgreSQL" icon="database">
    A complete PostgreSQL database with its own compute and storage, not just a gateway to Delta tables.
  </Card>

  <Card title="Serverless Scaling" icon="chart-line">
    Automatically scales compute based on workload. Supports scale-to-zero for cost efficiency.
  </Card>

  <Card title="Unity Catalog Integration" icon="link">
    Optionally register managed database catalogs to sync with Unity Catalog tables.
  </Card>

  <Card title="Standard Postgres" icon="code">
    Uses standard PostgreSQL syntax, permissions, and tools. Connect with any Postgres client.
  </Card>
</CardGroup>

## Security Architecture

<CardGroup cols={2}>
  <Card title="Data Encryption" icon="shield-halved">
    **At Rest:**

    * Database encrypted using industry-standard algorithms
    * Credentials encrypted and never exposed

    **In Transit:**

    * All traffic encrypted using TLS (sslmode=require)
    * Secure connection via PostgreSQL protocol
  </Card>

  <Card title="Access Control" icon="lock">
    **Authentication:**

    * Native PostgreSQL role with password
    * No token expiration (persistent connections)

    **Authorization:**

    * PostgreSQL role-based permissions
    * Managed at the Lakebase instance level
  </Card>
</CardGroup>

## Setting up Elementum Access in Databricks

### Prerequisites

Before starting:

1. Ensure you have **Workspace Admin** or **Account Admin** access
2. Have access to create Lakebase Postgres instances

### Setup Steps Overview

<Steps>
  <Step title="Create Lakebase Postgres Instance">
    Set up the managed PostgreSQL database.
  </Step>

  <Step title="Enable Native Postgres Login">
    Enable password-based authentication for Postgres roles.
  </Step>

  <Step title="Create Elementum Role">
    Create a Postgres role with password for Elementum to use.
  </Step>

  <Step title="Grant Permissions">
    Grant the role appropriate access to your data.
  </Step>

  <Step title="Create Platform Schema">
    Create an empty schema for Elementum platform operations.
  </Step>
</Steps>

## Run These Steps in Databricks

<Steps>
  <Step title="Create Lakebase Postgres Instance">
    Set up the managed PostgreSQL database:

    1. In your Databricks workspace, go to **Compute**
    2. Click the **Lakebase Postgres** tab
    3. Click **Create**
    4. Configure the instance:
       * **Name:** `elementum-lakebase`
       * **Instance size (Capacity Unit):** `2` (adjust based on workload)
       * **Serverless usage policy:** `None` (or configure as needed)
    5. Click **Create**
    6. Wait for the instance to show **Status: Available**
  </Step>

  <Step title="Enable Native Postgres Login">
    Enable password-based authentication for the Lakebase instance:

    1. On the Lakebase instance page, click **Edit** in the upper-right
    2. Turn on **Enable Postgres Native Role Login**
    3. Click **Save**

    <Callout type="info">
      This allows creating Postgres roles with passwords that don't expire, which is required for persistent connections like Elementum CloudLink.
    </Callout>
  </Step>

  <Step title="Get Connection Details">
    Note the connection parameters from the Lakebase instance:

    1. Click on the instance name (`elementum-lakebase`)
    2. Go to the **Connection details** tab
    3. Note the **Connection parameters:**
       * **host:** `instance-<uuid>.database.cloud.databricks.com`
       * **dbname:** `databricks_postgres` (default database)
       * **port:** `5432`

    <Callout type="info">
      The hostname format is: `instance-<instance-id>.database.cloud.databricks.com`
    </Callout>
  </Step>

  <Step title="Create Elementum Role with Password">
    Connect to the Lakebase instance via the **New Query** button or psql, and create a role for Elementum:

    ```sql theme={null}
    -- Create the Elementum role with a strong password
    CREATE ROLE elementum LOGIN PASSWORD 'your-strong-password-here';

    -- Grant database-level access
    GRANT ALL PRIVILEGES ON DATABASE databricks_postgres TO elementum;
    GRANT CONNECT ON DATABASE databricks_postgres TO elementum;
    ```

    <Warning>
      **Important:** Use a strong, unique password. Store it securely - you'll need it when configuring the Elementum CloudLink connection.
    </Warning>
  </Step>

  <Step title="Grant Permissions to Elementum Role">
    Grant the Elementum role appropriate access to your data:

    ### Choose Your Permission Level

    <Tabs>
      <Tab title="Superuser (Simplest)">
        For the simplest setup, grant superuser privileges:

        ```sql theme={null}
        -- Grant superuser role to elementum
        GRANT databricks_superuser TO elementum;
        ```

        <Callout type="warning">
          **Note:** This grants full access to all schemas and tables. Use specific permissions below if you need tighter access control.
        </Callout>
      </Tab>

      <Tab title="Specific Permissions (Recommended)">
        For fine-grained access control, grant specific PostgreSQL permissions:

        **Platform Schema Permissions (Required):**

        ```sql theme={null}
        -- Grant schema usage and full privileges
        GRANT USAGE ON SCHEMA elementum_platform TO elementum;
        GRANT ALL PRIVILEGES ON SCHEMA elementum_platform TO elementum;

        -- Grant privileges on all existing objects
        GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA elementum_platform TO elementum;
        GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA elementum_platform TO elementum;
        GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA elementum_platform TO elementum;

        -- Grant privileges on future objects (required for Elementum to create tables)
        ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
          GRANT ALL ON TABLES TO elementum;
        ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
          GRANT ALL ON SEQUENCES TO elementum;
        ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
          GRANT ALL ON FUNCTIONS TO elementum;
        ```

        **Data Schema Permissions (Choose based on your needs):**

        ```sql theme={null}
        -- Replace <schema_name> with your actual schema name

        -- Grant schema usage
        GRANT USAGE ON SCHEMA <schema_name> TO elementum;

        -- For read/write access to all tables in a schema:
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA <schema_name> 
          TO elementum;

        -- For read-only access to all tables in a schema:
        GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> 
          TO elementum;
        ```
      </Tab>
    </Tabs>

    ### Permission Examples

    <Tabs>
      <Tab title="Full Access (Read/Write)">
        ```sql theme={null}
        -- Grant full access to the sales schema
        GRANT USAGE ON SCHEMA sales TO elementum;
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA sales 
          TO elementum;

        -- Apply to future tables as well
        ALTER DEFAULT PRIVILEGES IN SCHEMA sales 
          GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO elementum;
        ```
      </Tab>

      <Tab title="Read-Only Access">
        ```sql theme={null}
        -- Grant read-only access to the analytics schema
        GRANT USAGE ON SCHEMA analytics TO elementum;
        GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO elementum;

        -- Apply to future tables as well
        ALTER DEFAULT PRIVILEGES IN SCHEMA analytics 
          GRANT SELECT ON TABLES TO elementum;
        ```
      </Tab>

      <Tab title="Specific Tables Only">
        ```sql theme={null}
        -- Grant access to specific tables only
        GRANT USAGE ON SCHEMA sales TO elementum;
        GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE sales.customers 
          TO elementum;
        GRANT SELECT ON TABLE sales.orders TO elementum;
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Create Platform Schema for Elementum">
    Create the platform schema and grant the Elementum role full access:

    ```sql theme={null}
    -- Create the platform schema (MUST BE EMPTY - for Elementum internal use only)
    CREATE SCHEMA IF NOT EXISTS elementum_platform;

    -- Grant the elementum role full access to the platform schema
    GRANT USAGE ON SCHEMA elementum_platform TO elementum;
    GRANT ALL PRIVILEGES ON SCHEMA elementum_platform TO elementum;

    -- Grant privileges on all existing objects
    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA elementum_platform TO elementum;
    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA elementum_platform TO elementum;
    GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA elementum_platform TO elementum;

    -- Grant privileges on future objects (required for Elementum to create tables)
    ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
      GRANT ALL ON TABLES TO elementum;
    ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
      GRANT ALL ON SEQUENCES TO elementum;
    ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
      GRANT ALL ON FUNCTIONS TO elementum;
    ```

    <Warning>
      **Critical:** The `elementum_platform` schema is for Elementum's internal operations. **Do NOT put your data tables here.** Your data tables should be in separate schemas.
    </Warning>
  </Step>

  <Step title="(Optional) Register Managed Database Catalog">
    If you want to access Unity Catalog tables through Lakebase:

    1. On the Lakebase instance page, go to the **Catalogs** tab
    2. Click **Create managed database catalog**
    3. Select the Unity Catalog you want to expose
    4. This creates a bridge between your Delta tables and the Postgres interface

    <Callout type="info">
      Managed database catalogs allow you to query Unity Catalog tables using standard PostgreSQL syntax through your Lakebase instance.
    </Callout>
  </Step>
</Steps>

## Configuring CloudLink in Elementum

After completing the Databricks setup, configure the connection in Elementum:

<Steps>
  <Step title="Navigate to CloudLink Settings">
    Go to **Settings > Cloud Links > Add Connection** and select **Databricks**.
  </Step>

  <Step title="Enter Connection Details">
    Fill in the connection form:

    | Field        | Value                                           | Description                                          |
    | ------------ | ----------------------------------------------- | ---------------------------------------------------- |
    | **Name**     | `Production Databricks`                         | Descriptive name for your connection                 |
    | **Hostname** | `instance-<uuid>.database.cloud.databricks.com` | Your Lakebase Postgres hostname                      |
    | **Port**     | `5432`                                          | PostgreSQL port                                      |
    | **Database** | `databricks_postgres`                           | The Lakebase database name                           |
    | **Schema**   | `elementum_platform`                            | **The empty platform schema (NOT your data schema)** |
    | **Username** | `elementum`                                     | The Postgres role name you created                   |
    | **Password** | `your-strong-password-here`                     | The password you set for the role                    |

    <Warning>
      **Critical - Schema Field:** Enter the empty platform schema you created (e.g., `elementum_platform`), NOT your data schema. If you enter your data schema here, it will be hidden from workflow building and you won't be able to access your data.
    </Warning>
  </Step>

  <Step title="Test the Connection">
    Click **Save** to test the connection. The system will verify:

    * Network connectivity
    * PostgreSQL authentication
    * Schema access

    <Callout type="success">
      If the connection saves successfully, your setup is complete.
    </Callout>
  </Step>

  <Step title="Configure Data Access">
    After saving the connection:

    1. **Select Tables:** Choose which tables to expose in Elementum
    2. **Configure Field Mapping:** Map columns to Elementum field types
    3. **Set Primary Key:** Identify the unique identifier column for each table
    4. **Configure Permissions:** Set which users/roles can access the data
  </Step>
</Steps>

## Verification and Testing

After completing the setup, verify everything is working:

<Steps>
  <Step title="Test Connection via psql">
    You can test the connection using the psql command from the Connection details tab:

    ```bash theme={null}
    psql "host=instance-<uuid>.database.cloud.databricks.com \
         user=<service-principal-uuid> \
         dbname=databricks_postgres \
         port=5432 \
         sslmode=require"
    ```

    When prompted for password, enter the password you set for the `elementum` role.
  </Step>

  <Step title="Verify Schema Access">
    Once connected, verify the platform schema exists:

    ```sql theme={null}
    \dn  -- List schemas

    -- Or:
    SELECT schema_name FROM information_schema.schemata;
    ```
  </Step>

  <Step title="Test in Elementum">
    1. Verify the connection shows as **Connected** in CloudLink settings
    2. Browse to the integrated tables in Elementum
    3. Verify data loads correctly
    4. Test creating/updating a record (if write access was granted)
  </Step>
</Steps>

## Troubleshooting

<Accordion title="Connection Issues">
  **Cannot Connect from Elementum:**

  * Verify the Lakebase instance is in "Available" status
  * Confirm the password is correct
  * Check that the Postgres role has been created and granted appropriate permissions
  * Verify the hostname is correct (should be `instance-<uuid>.database.cloud.databricks.com`)

  **"Authentication failed" Error:**

  * Verify the role name and password are correct
  * Ensure **Enable Postgres Native Role Login** is turned on
  * Confirm the role was created with `LOGIN` privilege
</Accordion>

<Accordion title="Cannot See Data Tables">
  **Tables not visible in Elementum:**

  * **Most common cause:** You entered your data schema in the Schema field instead of the platform schema
  * Verify the Postgres role has the appropriate permissions
  * Check that tables exist in the database
  * If using managed database catalogs, verify the catalog is registered
</Accordion>

<Accordion title="Permission Errors">
  **"Permission denied" Errors:**

  * Verify the Postgres role has appropriate GRANT permissions
  * Check that `databricks_superuser` or appropriate roles are granted
  * For specific table access, verify GRANT statements have been run
</Accordion>

<Accordion title="Lakebase Instance Issues">
  **Instance not starting:**

  * Check if you've reached capacity limits
  * Verify workspace has Lakebase Postgres enabled
  * Contact Databricks support if the instance stays in "Starting" state

  **Instance suspended:**

  * Lakebase instances can scale to zero when idle
  * The instance will automatically resume when a connection is made
  * First connection after suspension may take a few seconds
</Accordion>

## Security Best Practices

<CardGroup cols={2}>
  <Card title="Principle of Least Privilege" icon="shield">
    * Grant only necessary PostgreSQL roles
    * Use specific table grants instead of superuser where possible
    * Regularly audit granted permissions
    * Remove unused Postgres roles
  </Card>

  <Card title="Credential Security" icon="key">
    * Rotate passwords periodically (recommended: every 90 days)
    * Store passwords securely (never in code)
    * Use separate Postgres roles for different environments
    * Never share credentials outside authorized personnel
  </Card>

  <Card title="Network Security" icon="network-wired">
    * Lakebase uses TLS encryption by default (sslmode=require)
    * Consider workspace-level IP access controls
    * Monitor connection logs regularly
    * Set up alerts for suspicious activity
  </Card>

  <Card title="Monitoring" icon="chart-line">
    * Review query history via Lakebase Metrics tab
    * Monitor compute costs
    * Set up cost alerts in Databricks
    * Track data access patterns
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Configure Apps" icon="grid" href="/getting-started/fundamentals/core-concepts#processes--apps">
    Set up your first app in Elementum using your connected data
  </Card>

  <Card title="Create Automations" icon="bolt" href="../workflows/automation-system">
    Build workflows that act on your Databricks data
  </Card>

  <Card title="Setup AI Features" icon="sparkles" href="../ai-agents/ai-overview">
    Enable AI-powered search, automations, and insights
  </Card>

  <Card title="Data Best Practices" icon="lightbulb" href="/data/data-best-practices">
    Optimize your data models for Elementum
  </Card>
</CardGroup>

## Additional Resources

<CardGroup cols={2}>
  <Card title="CloudLink Overview" icon="cloud" href="/administration/setup-cloudlink">
    Learn more about CloudLink architecture
  </Card>

  <Card title="Databricks Documentation" icon="book" href="https://docs.databricks.com/aws/en/oltp/">
    Official Lakebase Postgres documentation
  </Card>

  <Card title="Get Support" icon="life-ring" href="../support/resources">
    Contact our team for setup assistance
  </Card>
</CardGroup>

***

*This guide reflects the latest Databricks Lakebase Postgres and Elementum best practices. For additional assistance, contact [support@elementum.io](mailto:support@elementum.io).*