## Connect Databricks to Lakehouse

URL: https://docs.atlan.com/platform/lakehouse/databricks/how-tos/connect-databricks

> Connect Databricks to Atlan's Lakehouse using foreign Iceberg tables in Unity Catalog to query metadata. Setup creates storage credentials and an external location pointing to Atlan's Lakehouse data in your cloud storage.

Databricks accesses Atlan's Lakehouse through foreign Iceberg tables in [Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/index.html) that point to Atlan's data in your cloud storage (S3 or ADLS). Setup requires creating storage credentials, an external location, and a foreign catalog in Databricks using the **Read Iceberg Via Metadata Location** private preview feature.

## Prerequisites

Before you begin, make sure that:

- You have permission in your Databricks workspace to create **storage credentials**, **external locations**, and **Unity Catalogs**.

- **AWS only:** During setup, you'll need to share your storage credential's **IAM Role ARN** and **External ID** with Atlan via a support ticket so Atlan can grant S3 access.

- The required Python dependency is installed automatically by the first cell of each notebook:

 ```python
 %pip install pyiceberg
 ```

## Enable Databricks private preview

Before setting up Databricks resources, in your Atlan workspace, navigate to **Workflow** > **Marketplace** > **Atlan Lakehouse** > **View connection details** and note your **catalog URI**, **catalog name**, **warehouse name**, **OAuth client ID**, and **OAuth client secret**. If you don't see them, contact [Atlan Support](https://docs.atlan.com/support/submit-request).

The foreign Iceberg tables workaround requires a Databricks Private Preview feature that must be enabled on your workspace before proceeding.

1. Contact your **Databricks account representative** and request enablement of the **Read Iceberg Via Metadata Location** Private Preview feature on your workspace.

2. Once your Databricks representative confirms the feature is enabled, notify [Atlan Support](https://docs.atlan.com/support/submit-request). Atlan uses this confirmation to prepare your storage access details and credentials. You'll be notified when ready—continue with [Set up Unity Catalog access](#set-up-unity-catalog-access) once Atlan confirms.

## Set up Unity Catalog access

Once Atlan provides your storage access details and credentials, follow the setup steps for your storage backend.

### AWS (S3)

Before proceeding, you must have received your **IAM Role ARN** and **Amazon S3 bucket path** from Atlan. If you haven't received them yet, open a ticket with [Atlan Support](https://docs.atlan.com/support/submit-request) and specify the tenant URL (for example, `yourcompany.atlan.com`) for which you need Lakehouse provisioned. Your **OAuth credentials** (Client ID and Client Secret) are available directly in the Atlan UI under **Workflows > Marketplace > Atlan Lakehouse > Connection Details**.

1. In your Databricks workspace, create a **storage credential** using the IAM Role ARN provided by Atlan:

 - Navigate to **Catalog Explorer > Credentials > Create Credential**.
 - Select **AWS IAM Role** as the credential type.
 - Enter the **IAM Role ARN** provided by Atlan.
 - In **Advanced Options**, enable **Limit to read-only use**.

2. Send the **IAM Role ARN** and **External ID** of the storage credential to [Atlan Support](https://docs.atlan.com/support/submit-request), along with your **Atlan tenant URL** and specify **Databricks** as the target compute engine. Atlan uses these to grant your credential read access to the S3 bucket. If you need multiple IAM roles (for example, for dev, test, and prod), include all ARNs in a single ticket. You'll be notified when access is granted before proceeding to step 3.

3. Once Atlan confirms access, create an **external location** in Unity Catalog pointing to the S3 path provided by Atlan:

 - Navigate to **Catalog Explorer > External Locations > Create External Location**.
 - Select **Manual** and choose **Amazon S3** as the storage type.
 - Enter the **Amazon S3 path** provided by Atlan.
 - Select the credential created in step 1.
 - Enable **Read-only** mode in advanced options, then click **Create**.

4. Create a **target catalog** where the foreign Iceberg tables are registered. This is the catalog name you configure as `DBX_CATALOG_NAME` in the scripts:

 - Navigate to **Catalog > Create a Catalog**.
 - Enter a catalog name and select **Standard** type.
 - For storage location, use a customer-managed storage location (not Atlan-managed).

### Azure (ADLS)

Atlan provides the following: Service Principal credentials (Directory/Tenant ID, Application/Client ID, Client Secret), Storage Account name, and OAuth credentials.

1. Create a **storage credential** in Unity Catalog using the Service Principal credentials provided by Atlan:

 - Navigate to **Catalog Explorer > Credentials > Create Credential**.
 - Select **Azure Service Principal** as the credential type.
 - Enter the **Directory (Tenant) ID**, **Application (Client) ID**, and **Client Secret** provided by Atlan.

### CLI

```bash
 databricks storage-credentials create --json '{
 "name": "<credential-name>",
 "azure_service_principal": {
 "directory_id": "",
 "application_id": "",
 "client_secret": ""
 }
 }'
 ```

2. Create an **external location** in Unity Catalog pointing to the ADLS path provided by Atlan:

 - Navigate to **Catalog Explorer > External Locations > Create External Location**.
 - Select **Manual** and choose **Azure Data Lake Storage** as the storage type.
 - Enter the ADLS path in the following format:
 ```
 abfss://objectstore@<storage-account-name>.dfs.core.windows.net/atlan-wh/
 ```

 :::note
 The final path segment is your Lakehouse catalog name (for example, `atlan-wh/` or `context_store/`). To find yours, go to `https://<domain-name>.atlan.com/workflows/setup/atlan-lakehouse` in your Atlan workspace.
 :::
 - Select the credential created in step 1.
 - In **Advanced Options**, enable **Limit to read-only use**.
 - Click **Test Connection** to validate, then click **Create**.

3. Create a **target catalog** where the foreign Iceberg tables are registered. This is the catalog name you configure as `DBX_CATALOG_NAME` in the scripts:

 - Navigate to **Catalog > Create a Catalog**.
 - Enter a catalog name and select **Standard** type.
 - Make sure the catalog storage is hosted on your own Azure tenant, not Atlan's.

## Create foreign Iceberg tables

Once Unity Catalog access is set up:

1. Download the create script [`dbx_foreign_iceberg_tables_create.py`](https://github.com/atlanhq/lakehouse-solutions/blob/main/databricks/foreign-iceberg-tables/dbx_foreign_iceberg_tables_create.py) from the Lakehouse Solutions repository and import it as a Databricks notebook. Set the following values in the **Configuration** cell:

 - `CLIENT_ID`: OAuth Client ID provided by Atlan
 - `CLIENT_SECRET`: OAuth Client Secret provided by Atlan
 - `POLARIS_CATALOG_URI`: Catalog URI provided by Atlan (for example, `https://<tenant>.atlan.com/api/polaris/api/catalog`)
 - `CATALOG_NAME`: Polaris catalog name provided by Atlan
 - `WAREHOUSE_NAME`: Polaris warehouse name provided by Atlan
 - `DBX_CATALOG_NAME`: Target Unity Catalog name created in the previous section
 - `HISTORY_NAMESPACE_SYNC`: Set to `true` to include the `atlan-history` namespace (default: `false`)

2. Run the notebook. The script autodetects the Polaris warehouse, discovers all namespaces and tables, and creates schemas and foreign Iceberg tables in the target Unity Catalog. The script uses `CREATE TABLE IF NOT EXISTS`, making it safe to re-run.

 *Example:* To verify the setup and query metadata for assets registered in Atlan:

 ```sql
 SELECT *
 FROM .gold.assets
 LIMIT 10;
 ```

## Refresh foreign Iceberg tables

Foreign Iceberg tables don't sync automatically. Run the refresh script periodically to keep tables up to date with the latest Lakehouse data.

1. Download the refresh script [`dbx_foreign_iceberg_tables_refresh.py`](https://github.com/atlanhq/lakehouse-solutions/blob/main/databricks/foreign-iceberg-tables/dbx_foreign_iceberg_tables_refresh.py) from the Lakehouse Solutions repository and import it as a Databricks notebook. Configure the same variables as the create script.

2. Schedule the notebook to run on a recurring basis:

 - **Maximum frequency**: No more than once every 30 minutes

 The refresh script uses `REFRESH TABLE` to update metadata pointers without recreating tables.

## Troubleshooting

If you have any issues configuring or querying foreign Iceberg tables in Databricks, see [Troubleshooting Databricks errors](https://docs.atlan.com/llms/governance/lakehouse/databricks/llms.txt).

## Next steps

Now that Databricks is connected to Lakehouse, you can:

- **Query Atlan metadata from Databricks**: See the available metadata tables in [Entity metadata reference](https://docs.atlan.com/llms/governance/lakehouse/entity-metadata/llms.txt).
- **Use cases**: Explore popular patterns such as metadata enrichment tracking, lineage impact analysis, and glossary alignment in [Use cases](https://docs.atlan.com/llms/governance/lakehouse/overview/llms.txt).

---