## Crawl Databricks

URL: https://docs.atlan.com/apps/connectors/data-warehouses/databricks/how-tos/crawl-databricks

> Crawl metadata from Databricks to catalog tables, schemas, views, and volumes. Extract lineage and column information after configuring access permissions and authentication.

Atlan crawls your Databricks workspace to catalog tables, schemas, views, and volumes, and extracts column-level lineage automatically.

Extract metadata from your [Databricks](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt) workspace into Atlan to catalog tables, schemas, views, and volumes. After configuring the necessary access permissions, you can establish a connection and crawl to extract lineage and column-level details. If you use [AWS PrivateLink](https://docs.atlan.com/llms/connectors/databricks/aws-private-link-to-databricks/llms.txt) or [Azure Private Link](https://docs.atlan.com/llms/connectors/databricks/azure-private-link-to-databricks/llms.txt) for network security, complete that setup first. Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting.

## Select source

To select Databricks as your source:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Databricks** and select the **Databricks** tile by Atlan, which extracts metadata from Unity Catalog. Don't select the **Databricks Miner** tiles, which are used for [lineage and usage extraction](https://docs.atlan.com/llms/connectors/databricks/extract-lineage-and-usage-from-databricks/llms.txt).

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

## Provide credentials

Choose your extraction method:

:::warning Offline extraction sunset
The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt).
:::

- In **Direct** extraction, Atlan connects to your database and crawls metadata directly. Next, select an authentication method:
 - In **JDBC**, you need a [personal access token and HTTP path for authentication](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).
 - In **AWS Service**, you need a [client ID and client secret for AWS service principal authentication](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).
 - In **Azure Service**, you need a [tenant ID, client ID, and client secret for Azure service principal authentication](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).
- In **Agent** extraction, Atlan's secure agent executes metadata extraction within the organization's environment.

### Direct extraction method

#### JDBC

To enter your Databricks credentials:

1. For _Host_, enter the hostname, [AWS PrivateLink endpoint](https://docs.atlan.com/llms/connectors/databricks/aws-private-link-to-databricks/llms.txt), or [Azure Private Link endpoint](https://docs.atlan.com/llms/connectors/databricks/azure-private-link-to-databricks/llms.txt) for your Databricks instance.

2. For _Port_, enter the port number of your Databricks instance.

3. For _Personal Access Token_, enter the access token you generated when [setting up access](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

4. For _HTTP Path_, enter one of the following:
 - A path starting with `/sql/1.0/warehouses` to use the [Databricks SQL warehouse](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).
 - A path starting with `sql/protocolv1/o` to use the [Databricks interactive cluster](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).
5. Click **Test Authentication** to confirm connectivity to Databricks using these details.

6. Once successful, at the bottom of the screen click **Next**.

:::warning
Make sure your Databricks instance (SQL warehouse or interactive cluster) is up and running, otherwise the **Test Authentication** step times out.
:::

#### AWS service principal

To enter your Databricks credentials:

1. For _Host_, enter the hostname or [AWS PrivateLink endpoint](https://docs.atlan.com/llms/connectors/databricks/aws-private-link-to-databricks/llms.txt) for your Databricks instance.

2. For _Port_, enter the port number of your Databricks instance.

3. For _Client ID_, enter the [client ID for your AWS service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

4. For _Client Secret_, enter the [client secret for your AWS service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

5. Click **Test Authentication** to confirm connectivity to Databricks using these details.

6. Once successful, at the bottom of the screen click **Next**.

#### Azure service principal

To enter your Databricks credentials:

1. For _Host_, enter the hostname or [Azure Private Link endpoint](https://docs.atlan.com/llms/connectors/databricks/azure-private-link-to-databricks/llms.txt) for your Databricks instance.

2. For _Port_, enter the port number of your Databricks instance.

3. For _Client ID_, enter the [application (client) ID for your Azure service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

4. For _Client Secret_, enter the [client secret for your Azure service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

5. For _Tenant ID_, enter the [directory (tenant) ID for your Azure service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

6. Click **Test Authentication** to confirm connectivity to Databricks using these details.

7. Once successful, at the bottom of the screen click **Next**.

### Agent extraction method

Atlan supports using a Secure Agent for fetching metadata from Databricks. To use a Secure Agent, follow these steps:

1. Select the **Agent** tab.

2. Configure the Databricks data source by adding the secret keys for your secret store. For details on the required fields, refer to the Direct extraction section.

3. Complete the Secure Agent configuration by following the instructions in the [How to configure Secure Agent for workflow execution](https://docs.atlan.com/llms/platform/secure-agent/configure-secure-agent-for-workflow-execution/llms.txt) guide.

4. Click **Next** after completing the configuration.

## Configure connection

To complete the Databricks connection configuration:

1. Provide a _Connection Name_ that represents your source environment. For example, you might want to use values like `production`, `development`, `gold`, or `analytics`.

2. (Optional) To change the users able to manage this connection, change the users or groups listed under _Connection Admins_.

 :::warning
 If you don't specify any user or group, nobody can manage the connection - not even admins.
 :::

3. (Optional) To prevent users from querying any Databricks data, change _Enable SQL Query_ to **No**.

4. (Optional) To prevent users from previewing any Databricks data, change _Enable Data Preview_ to **No**.

5. (Optional) To prevent users from running large queries, change _Max Row Limit_ or keep the default selection.

6. At the bottom of the screen, click the **Next** button to proceed.

## Configure crawler

Before running the Databricks crawler, you can further configure it.

### System tables extraction method

The system metadata extraction method is only available for [Unity Catalog-enabled workspaces](https://docs.databricks.com/data-governance/unity-catalog/get-started.html). It provides access to detailed metadata from system tables and supports all three authentication types. You can extract metadata from your Databricks workspace using this method. Follow these steps:

1. Set up authentication using one of the following:
 - [Personal access token](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt)
 - [AWS service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt)
 - [Azure service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt)

2. The default options can work as is. You may choose to override the defaults for any of the remaining options:

 - For _Asset selection_, select a filtering option:
 - For _SQL warehouse_, click the dropdown to select the SQL warehouse you want to configure.
 - To select the assets you want to include in crawling, click **Include by hierarchy** and filter for assets down to the database or schema level. (This defaults to all assets, if none are specified.)
 - To have the crawler include _Databases_, _Schemas_, or _Tables & Views_ based on a naming convention, click **Include by regex** and specify a regular expression - for example, specifying `ATLAN_EXAMPLE_DB.*` for _Databases_ includes all the matching databases and their child assets.
 - To select the assets you want to exclude from crawling, click **Exclude by hierarchy** and filter for assets down to the database or schema level. (This defaults to no assets, if none are specified.) 
 - To have the crawler ignore _Databases_, _Schemas_, or _Tables & Views_ based on a naming convention, click **Exclude by regex** and specify a regular expression - for example, specifying `ATLAN_EXAMPLE_TABLES.*` for _Tables & Views_ excludes all the matching tables and views.
 - Click **+** to add more filters. If you add multiple filters, assets are crawled based on matching _all_ the filtering conditions you have set.
 - To [import tags from Databricks to Atlan](https://docs.atlan.com/llms/connectors/databricks/manage-databricks-tags/llms.txt), change _Import Tags_ to **Yes**. Note that you must have a [Unity Catalog-enabled workspace](https://docs.databricks.com/en/data-governance/unity-catalog/get-started.html) to import Databricks tags in Atlan.

 :::info **Did you know?** 
 If an asset appears in both the include and exclude filters, the exclude filter takes precedence.
 :::

 - To catalog [Databricks AI/BI dashboards](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks-ai-bi-dashboards/llms.txt), turn on _Import AI/BI Dashboards_. This option is available only with the system tables extraction method, and only when the private preview is enabled for your tenant.

#### Incremental extraction 

 - Toggle incremental extraction, for a faster and more efficient metadata extraction.

### JDBC extraction method

The JDBC extraction method uses JDBC queries to extract metadata from your Databricks instance. This was the original extraction method provided by Databricks. This extraction method is only supported for [personal access token authentication](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

You can override the defaults for any of these options:

- To select the assets you want to include in crawling, click **Include Metadata**. (This defaults to all assets, if none are specified.)
- To select the assets you want to exclude from crawling, click **Exclude Metadata**. (This defaults to no assets if none are specified.)
- To have the crawler ignore tables and views based on a naming convention, specify a regular expression in the _Exclude regex for tables & views_ field.
- For _View Definition Lineage_, keep the default **Yes** to generate upstream lineage for views based on the tables referenced in the views or click **No** to exclude from crawling.
- For _Advanced Config_, keep _Default_ for the default configuration or click **Advanced** to further configure the crawler:
 - To enable or disable schema-level filtering at source, click **Enable Source Level Filtering** and select **True** to enable it or **False** to disable it.

### REST API extraction method

:::danger Deprecated
The REST API extraction method for crawling Databricks metadata is **deprecated**. Use the [system tables extraction method](#system-tables-extraction-method) instead, which provides more reliable and comprehensive metadata extraction. The REST API extraction method is scheduled for removal in a future release.
:::

The REST API extraction method uses [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/index.html) to extract metadata from your Databricks instance. This extraction method is supported for all three authentication options: [personal access token](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt), [AWS service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt), and [Azure service principal](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

- This method is only supported by [Unity Catalog-enabled](https://docs.databricks.com/data-governance/unity-catalog/get-started.html) workspaces.
- If you enable an existing workspace, you also need to [upgrade your tables and views to Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/migrate.html).

While REST APIs are used to extract metadata, JDBC queries are still used for querying purposes.

You can override the defaults for any of these options:

- Change the extraction method under _Extraction method_ to **REST API**.
- For _Asset selection_, select a filtering option:
 - To select the databases you want to include in crawling, click **Include by hierarchy** and select the databases. (This defaults to all databases, if none are specified.)
 - To have the crawler include _Tables & Views_ based on a naming convention, click **Include by regex** and specify a regular expression - for example, specifying `ATLAN_EXAMPLE_TABLES.*` includes all the matching tables and views.
 - To select the databases you want to exclude from crawling, click **Exclude by hierarchy** and select the databases. (This defaults to no databases, if none are specified.)
 - To have the crawler ignore _Tables & Views_ based on a naming convention, click **Exclude by regex** and specify a regular expression - for example, specifying `ATLAN_EXAMPLE_TABLES.*` excludes all the matching tables and views.
 - Click **+** to add more filters. If you add multiple filters, assets are crawled based on matching _all_ the filtering conditions you have set.
 - To [import tags from Databricks to Atlan](https://docs.atlan.com/llms/connectors/databricks/manage-databricks-tags/llms.txt), change _Import Tags_ to **Yes**. Note that you must have a [Unity Catalog-enabled workspace](https://docs.databricks.com/en/data-governance/unity-catalog/get-started.html) to import Databricks tags in Atlan.

 :::info **Did you know?**
 If an asset appears in both the include and exclude filters, the exclude filter takes precedence.
 :::

## Run crawler

Follow these steps to run the Databricks crawler:

1. To check for any [permissions or other configuration issues](https://docs.atlan.com/llms/connectors/databricks/preflight-checks-for-databricks/llms.txt) before running the crawler, click **Preflight checks**.

2. You can either:
 - To run the crawler once immediately, at the bottom of the screen, click the **Run** button.
 - To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule Run** button.

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

---