Connecting Databricks to Elementum - Elementum Documentation

Get your CloudLink connection up and running with Databricks Lakebase Postgres. This guide covers setting up a native PostgreSQL role and connecting via the managed PostgreSQL database.

Secure Direct Access

Elementum provides secure, in-place data access to your Databricks Lakebase Postgres database:

Authentication

Native PostgreSQL role with password authenticates Elementum. You retain full control over the data and can terminate access at any time.

In-Place Access

Data stays in your Databricks instance. No data is copied or moved to external systems.

How It Works

Databricks Side

A Lakebase Postgres instance provides a managed PostgreSQL database. Service principal permissions control access to the database.

Elementum Side

Elementum connects via standard PostgreSQL protocol (port 5432) using native Postgres credentials. Further access restrictions are applied through Elementum access policies.

What is Lakebase Postgres?

Lakebase Postgres is a fully managed, cloud-native PostgreSQL database within Databricks. Key features:

Managed PostgreSQL

A complete PostgreSQL database with its own compute and storage, not just a gateway to Delta tables.

Serverless Scaling

Automatically scales compute based on workload. Supports scale-to-zero for cost efficiency.

Unity Catalog Integration

Optionally register managed database catalogs to sync with Unity Catalog tables.

Standard Postgres

Uses standard PostgreSQL syntax, permissions, and tools. Connect with any Postgres client.

Security Architecture

Data Encryption

At Rest:

Database encrypted using industry-standard algorithms
Credentials encrypted and never exposed

In Transit:

All traffic encrypted using TLS (sslmode=require)
Secure connection via PostgreSQL protocol

Access Control

Authentication:

Native PostgreSQL role with password
No token expiration (persistent connections)

Authorization:

PostgreSQL role-based permissions
Managed at the Lakebase instance level

Setting up Elementum Access in Databricks

Prerequisites

Before starting:

Ensure you have Workspace Admin or Account Admin access
Have access to create Lakebase Postgres instances
Have access to create Lakebase Postgres instances

Setup Steps Overview

Create Lakebase Postgres Instance

Set up the managed PostgreSQL database.

Enable Native Postgres Login

Enable password-based authentication for Postgres roles.

Create Elementum Role

Create a Postgres role with password for Elementum to use.

Grant Permissions

Grant the role appropriate access to your data.

Create Platform Schema

Create an empty schema for Elementum platform operations.

Run These Steps in Databricks

Create Lakebase Postgres Instance

Set up the managed PostgreSQL database:

In your Databricks workspace, go to Compute
Click the Lakebase Postgres tab
Click Create
Configure the instance:
- Name: elementum-lakebase
- Instance size (Capacity Unit): 2 (adjust based on workload)
- Serverless usage policy: None (or configure as needed)
Click Create
Wait for the instance to show Status: Available

Enable Native Postgres Login

Enable password-based authentication for the Lakebase instance:

On the Lakebase instance page, click Edit in the upper-right
Turn on Enable Postgres Native Role Login
Click Save

This allows creating Postgres roles with passwords that don’t expire, which is required for persistent connections like Elementum CloudLink.

Get Connection Details

Note the connection parameters from the Lakebase instance:

Click on the instance name (elementum-lakebase)
Go to the Connection details tab
Note the Connection parameters:
- host: instance-<uuid>.database.cloud.databricks.com
- dbname: databricks_postgres (default database)
- port: 5432

The hostname format is: instance-<instance-id>.database.cloud.databricks.com

Create Elementum Role with Password

Connect to the Lakebase instance via the New Query button or psql, and create a role for Elementum:

-- Create the Elementum role with a strong password
CREATE ROLE elementum LOGIN PASSWORD 'your-strong-password-here';

-- Grant database-level access
GRANT ALL PRIVILEGES ON DATABASE databricks_postgres TO elementum;
GRANT CONNECT ON DATABASE databricks_postgres TO elementum;

Important: Use a strong, unique password. Store it securely - you’ll need it when configuring the Elementum CloudLink connection.

Grant Permissions to Elementum Role

Grant the Elementum role appropriate access to your data:

Choose Your Permission Level

Superuser (Simplest)
Specific Permissions (Recommended)

For the simplest setup, grant superuser privileges:

-- Grant superuser role to elementum
GRANT databricks_superuser TO elementum;

Note: This grants full access to all schemas and tables. Use specific permissions below if you need tighter access control.

For fine-grained access control, grant specific PostgreSQL permissions:Platform Schema Permissions (Required):

-- Grant schema usage and full privileges
GRANT USAGE ON SCHEMA elementum_platform TO elementum;
GRANT ALL PRIVILEGES ON SCHEMA elementum_platform TO elementum;

-- Grant privileges on all existing objects
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA elementum_platform TO elementum;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA elementum_platform TO elementum;
GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA elementum_platform TO elementum;

-- Grant privileges on future objects (required for Elementum to create tables)
ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
  GRANT ALL ON TABLES TO elementum;
ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
  GRANT ALL ON SEQUENCES TO elementum;
ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
  GRANT ALL ON FUNCTIONS TO elementum;

Data Schema Permissions (Choose based on your needs):

-- Replace <schema_name> with your actual schema name

-- Grant schema usage
GRANT USAGE ON SCHEMA <schema_name> TO elementum;

-- For read/write access to all tables in a schema:
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA <schema_name> 
  TO elementum;

-- For read-only access to all tables in a schema:
GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> 
  TO elementum;

Permission Examples

Full Access (Read/Write)
Read-Only Access
Specific Tables Only

-- Grant full access to the sales schema
GRANT USAGE ON SCHEMA sales TO elementum;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA sales 
  TO elementum;

-- Apply to future tables as well
ALTER DEFAULT PRIVILEGES IN SCHEMA sales 
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO elementum;

-- Grant read-only access to the analytics schema
GRANT USAGE ON SCHEMA analytics TO elementum;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO elementum;

-- Apply to future tables as well
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics 
  GRANT SELECT ON TABLES TO elementum;

-- Grant access to specific tables only
GRANT USAGE ON SCHEMA sales TO elementum;
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE sales.customers 
  TO elementum;
GRANT SELECT ON TABLE sales.orders TO elementum;

Create Platform Schema for Elementum

Create the platform schema and grant the Elementum role full access:

-- Create the platform schema (MUST BE EMPTY - for Elementum internal use only)
CREATE SCHEMA IF NOT EXISTS elementum_platform;

-- Grant the elementum role full access to the platform schema
GRANT USAGE ON SCHEMA elementum_platform TO elementum;
GRANT ALL PRIVILEGES ON SCHEMA elementum_platform TO elementum;

-- Grant privileges on all existing objects
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA elementum_platform TO elementum;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA elementum_platform TO elementum;
GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA elementum_platform TO elementum;

-- Grant privileges on future objects (required for Elementum to create tables)
ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
  GRANT ALL ON TABLES TO elementum;
ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
  GRANT ALL ON SEQUENCES TO elementum;
ALTER DEFAULT PRIVILEGES IN SCHEMA elementum_platform 
  GRANT ALL ON FUNCTIONS TO elementum;

Critical: The elementum_platform schema is for Elementum’s internal operations. Do NOT put your data tables here. Your data tables should be in separate schemas.

(Optional) Register Managed Database Catalog

If you want to access Unity Catalog tables through Lakebase:

On the Lakebase instance page, go to the Catalogs tab
Click Create managed database catalog
Select the Unity Catalog you want to expose
This creates a bridge between your Delta tables and the Postgres interface

Managed database catalogs allow you to query Unity Catalog tables using standard PostgreSQL syntax through your Lakebase instance.

Configuring CloudLink in Elementum

After completing the Databricks setup, configure the connection in Elementum:

Navigate to CloudLink Settings

Go to Settings > Cloud Links > Add Connection and select Databricks.

Enter Connection Details

Fill in the connection form:

Field	Value	Description
Name	`Production Databricks`	Descriptive name for your connection
Hostname	`instance-<uuid>.database.cloud.databricks.com`	Your Lakebase Postgres hostname
Port	`5432`	PostgreSQL port
Database	`databricks_postgres`	The Lakebase database name
Schema	`elementum_platform`	The empty platform schema (NOT your data schema)
Username	`elementum`	The Postgres role name you created
Password	`your-strong-password-here`	The password you set for the role

Critical - Schema Field: Enter the empty platform schema you created (e.g., elementum_platform), NOT your data schema. If you enter your data schema here, it will be hidden from workflow building and you won’t be able to access your data.

Test the Connection

Click Save to test the connection. The system will verify:

Network connectivity
PostgreSQL authentication
Schema access

If the connection saves successfully, your setup is complete.

Configure Data Access

After saving the connection:

Select Tables: Choose which tables to expose in Elementum
Configure Field Mapping: Map columns to Elementum field types
Set Primary Key: Identify the unique identifier column for each table
Configure Permissions: Set which users/roles can access the data

Verification and Testing

After completing the setup, verify everything is working:

Test Connection via psql

You can test the connection using the psql command from the Connection details tab:

psql "host=instance-<uuid>.database.cloud.databricks.com \
     user=<service-principal-uuid> \
     dbname=databricks_postgres \
     port=5432 \
     sslmode=require"

When prompted for password, enter the password you set for the elementum role.

Verify Schema Access

Once connected, verify the platform schema exists:

\dn  -- List schemas

-- Or:
SELECT schema_name FROM information_schema.schemata;

Test in Elementum

Verify the connection shows as Connected in CloudLink settings
Browse to the integrated tables in Elementum
Verify data loads correctly
Test creating/updating a record (if write access was granted)

Troubleshooting

Connection Issues

Cannot Connect from Elementum:

Verify the Lakebase instance is in “Available” status
Confirm the password is correct
Check that the Postgres role has been created and granted appropriate permissions
Verify the hostname is correct (should be instance-<uuid>.database.cloud.databricks.com)

“Authentication failed” Error:

Verify the role name and password are correct
Ensure Enable Postgres Native Role Login is turned on
Confirm the role was created with LOGIN privilege

Cannot See Data Tables

Tables not visible in Elementum:

Most common cause: You entered your data schema in the Schema field instead of the platform schema
Verify the Postgres role has the appropriate permissions
Check that tables exist in the database
If using managed database catalogs, verify the catalog is registered

Permission Errors

“Permission denied” Errors:

Verify the Postgres role has appropriate GRANT permissions
Check that databricks_superuser or appropriate roles are granted
For specific table access, verify GRANT statements have been run

Lakebase Instance Issues

Instance not starting:

Check if you’ve reached capacity limits
Verify workspace has Lakebase Postgres enabled
Contact Databricks support if the instance stays in “Starting” state

Instance suspended:

Lakebase instances can scale to zero when idle
The instance will automatically resume when a connection is made
First connection after suspension may take a few seconds

Security Best Practices

Principle of Least Privilege

Grant only necessary PostgreSQL roles
Use specific table grants instead of superuser where possible
Regularly audit granted permissions
Remove unused Postgres roles

Credential Security

Rotate passwords periodically (recommended: every 90 days)
Store passwords securely (never in code)
Use separate Postgres roles for different environments
Never share credentials outside authorized personnel

Network Security

Lakebase uses TLS encryption by default (sslmode=require)
Consider workspace-level IP access controls
Monitor connection logs regularly
Set up alerts for suspicious activity

Monitoring

Review query history via Lakebase Metrics tab
Monitor compute costs
Set up cost alerts in Databricks
Track data access patterns

Next Steps

Configure Apps

Set up your first app in Elementum using your connected data

Create Automations

Build workflows that act on your Databricks data

Setup AI Features

Enable AI-powered search, automations, and insights

Data Best Practices

Optimize your data models for Elementum

Additional Resources

CloudLink Overview

Learn more about CloudLink architecture

Databricks Documentation

Official Lakebase Postgres documentation

Get Support

Contact our team for setup assistance

This guide reflects the latest Databricks Lakebase Postgres and Elementum best practices. For additional assistance, contact support@elementum.io.

Getting Started

Support

​Secure Direct Access

Authentication

In-Place Access

​How It Works

​What is Lakebase Postgres?

Managed PostgreSQL

Serverless Scaling

Unity Catalog Integration

Standard Postgres

​Security Architecture

Data Encryption

Access Control

​Setting up Elementum Access in Databricks

​Prerequisites

​Setup Steps Overview

​Run These Steps in Databricks

​Choose Your Permission Level

​Permission Examples

​Configuring CloudLink in Elementum

​Verification and Testing

​Troubleshooting

​Security Best Practices

Principle of Least Privilege

Credential Security

Network Security

Monitoring

​Next Steps

Configure Apps

Create Automations

Setup AI Features

Data Best Practices

​Additional Resources

CloudLink Overview

Databricks Documentation

Get Support

Secure Direct Access

How It Works

What is Lakebase Postgres?

Security Architecture

Setting up Elementum Access in Databricks

Prerequisites

Setup Steps Overview

Run These Steps in Databricks

Choose Your Permission Level

Permission Examples

Configuring CloudLink in Elementum

Verification and Testing

Troubleshooting

Security Best Practices

Next Steps

Additional Resources