## Set up Databricks

URL: https://docs.atlan.com/apps/connectors/data-warehouses/databricks/how-tos/set-up-databricks

> Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods:.

Atlan connects to Databricks using any of three authentication methods to crawl Unity Catalog metadata and build lineage.

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods:

### Personal access token authentication

### Personal access token authentication

:::warning Who can do this?
 Check that you have _Admin_ and _Databricks SQL access_ for the Databricks workspace. This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator.
:::

### Grant user access to workspace

To grant workspace access to the user creating a personal access token:

1. From the left menu of the account console, click **Workspaces** and then select a workspace to which you want to add the user.
2. From the tabs along the top of your workspace page, click the **Permissions** tab.
3. In the upper right of the _Permissions_ page, click **Add permissions**.
4. In the _Add permissions_ dialog, enter the following details:
 1. For _User, group, or service principal_, select the user to grant access.
 2. For _Permission_, click the dropdown and select workspace **User.**

### Generate a personal access token

You can [generate a personal access token](https://docs.databricks.com/en/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users) in your Databricks workspace to the authenticate the [integration in Atlan](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks/llms.txt).

To generate a personal access token:

1. From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click **User** **Settings**.
2. Under the _Settings_ menu, click **Developer**.
3. On the _Developer_ page, next to _Access tokens_, click **Manage**.
4. On the _Access tokens_ page, click the **Generate new token** button.
5. In the _Generate new token_ dialog:
 1. For _Comment_, enter a description of the token's intended use - for example, `Atlan crawler`.
 2. For _Lifetime (days)_, consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed.

 :::warning Important!
 If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.
 :::

 3. At the bottom of the dialog, click **Generate**.
6. Copy and save the generated token in a secure location, and then click **Done**.

### Select a cluster

:::info **Did you know?** 
 Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to [Databricks documentation](https://docs.databricks.com/en/admin/sql/serverless.html) for AWS Databricks workspaces or [Microsoft documentation](https://learn.microsoft.com/en-us/azure/databricks/admin/sql/serverless) for Azure Databricks workspaces.

:::

You can set up personal access token authentication for your Databricks instance using one of the following cluster options:

- Interactive cluster
- SQL warehouse (formerly SQL endpoint)

#### Interactive cluster

To confirm an [all-purpose interactive cluster](https://docs.databricks.com/clusters/index.html) is configured:

1. From the left menu of any page of your Databricks instance, click **Compute**.
2. Under the _All-purpose clusters_ tab, verify you have a cluster defined.
3. Click the link under the _Name_ column of the table to open your cluster.
4. Under the _Configuration_ tab, verify the _Autopilot options_ to _Terminate after ... minutes_ is enabled.
5. At the bottom of the _Configuration_ tab, expand the **Advanced options** expandable.
 1. Under the _Advanced options_ expandable, open the **JDBC/ODBC** tab.
 2. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: _Server Hostname_, _Port_, and _HTTP Path_.

#### SQL warehouse (formerly SQL endpoint)

To confirm a [SQL warehouse](https://docs.databricks.com/en/compute/sql-warehouse/index.html) is configured:

1. From the left menu of any page of your Databricks instance, open the dropdown just below the _databricks_ logo and change to **SQL**.
2. From the refreshed left menu, click **SQL Warehouses**.
3. Click the link under the _Name_ column of the table to open your SQL warehouse.
4. Under the **Connection details** tab, confirm that all of the fields are populated and copy them for use in crawling: _Server hostname_, _Port_, and _HTTP path_.

### AWS service principal authentication

### AWS service principal authentication

:::warning Who can do this?
 You need your [AWS Databricks account admin](https://docs.databricks.com/en/administration-guide/users-groups/service-principals.html#who-can-manage-and-use-service-principals) to create a service principal and manage OAuth credentials for the service principal and your [AWS Databricks workspace admin](https://docs.databricks.com/en/administration-guide/users-groups/service-principals.html#who-can-manage-and-use-service-principals) to add the service principal to your AWS Databricks workspace - you may not have access yourself.
:::

You need the following to authenticate the connection in Atlan:

- Client ID
- Client secret

### Create a service principal

You can create a service principal directly in your Databricks account or from a Databricks workspace.

- Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces.
- Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace.

#### Identity federation enabled

To create a service principal from your Databricks account, with identify federation enabled:

1. Log in to your Databricks [account console](https://accounts.cloud.databricks.com/) as an account admin.
2. From the left menu of the account console, click **User management**.
3. From the tabs along the top of the _User management_ page, click the **Service principals** tab.
4. In the upper right of the _Service principals_ page, click **Add service principal**.
5. On the _Add service principal_ page, enter a name for the service principal and then click **Add**.
6. Once the service principal has been created, you can assign it to your identity federated workspace. From the left menu of the account console, click **Workspaces** and then select a workspace to which you want to add the service principal.
7. From the tabs along the top of your workspace page, click the **Permissions** tab.
8. In the upper right of the _Permissions_ page, click **Add permissions**.
9. In the _Add permissions_ dialog, enter the following details:
 1. For _User, group, or service principal_, select the service principal you created.
 2. For _Permission_, click the dropdown and select workspace **User.**

#### Identity federation disabled

To create a service principal from a Databricks workspace, with identity federation disabled:

1. Log in to your AWS Databricks workspace as a workspace admin.
2. From the top right of your workspace, click your username, and then from the dropdown, click **Admin Settings**.
3. In the left menu of the _Settings_ page, under the _Workspace admin_ subheading, click **Identity and access**.
4. On the _Identity and access_ page, under _Management and permissions_, next to _Service principals_, click **Manage**. 
5. In the upper right of the _Service principals_ page, click **Add service principal**.
6. In the _Add service principal_ dialog, click the **Add new** button.
7. For _New service principal display name_, enter a name for the service principal and then click **Add**.

### Create an OAuth secret for the service principal

You need to create an OAuth secret to authenticate to Databricks REST APIs.

To create an OAuth secret for the [service principal](#create-a-service-principal):

1. Log in to your Databricks [account console](https://accounts.cloud.databricks.com/) as an account admin.
2. From the left menu of the account console, click **User management**.
3. From the tabs along the top of the _User management_ page, click the **Service principals** tab.
4. In the upper right of the _Service principals_ page, select the [service principal you created](#create-a-service-principal).
5. On the service principal page, under _OAuth secrets_, click **Generate secret**.
6. From the _Generate secret_ dialog, copy the _Secret_ and _Client ID_ and store it in a secure location.

 :::warning Important!
 Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal.
 :::

7. Once you've copied the client ID and secret, click **Done**.

### Azure service principal authentication

### Azure service principal authentication

:::warning Who can do this?
 You need your [Azure Databricks account admin](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/service-principals#--who-can-manage-and-use-service-principals) to create a service principal and your [Azure Databricks workspace admin](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/service-principals#--who-can-manage-and-use-service-principals) to add the service principal to your Azure Databricks workspace - you may not have access yourself.
:::

You need the following to authenticate the connection in Atlan:

- Client ID (application ID)
- Client secret
- Tenant ID (directory ID)

### Create a service principal

To [use service principals on Azure Databricks](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/service-principals#--manage-service-principals-in-your-account), an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal.

To create a service principal:

1. Sign in to the [Azure portal](https://portal.azure.com/).
2. If you have access to multiple tenants, subscriptions, or directories, click the **Directories + subscriptions** (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal.
3. In_Search resources, services, and docs_, search for and select**Microsoft Entra ID**.
4. Click**\+ Add**and select**App registration**.
5. For_Name_, enter a name for the application.
6. In the_Supported account types_section, select**Accounts in this organizational directory only (Single tenant)** and then click **Register**.
7. On the application page's_Overview_page, in the_Essentials_section, copy and store the following values in a secure location:
 - _Application (client) ID_
 - _Directory (tenant) ID_
8. To generate a client secret, within_Manage_, click**Certificates & secrets**.
9. On the_Client secrets_tab, click**New client secret**.
10. In the_Add a client secret_dialog, enter the following details:
 1. For _Description_, enter a description for the client secret.
 2. For_Expires_, select an expiry time period for the client secret and then click **Add**.
 3. Copy and store the client secret's_Value_in a secure place.

### Add a service principal to your account

To add a service principal to your Azure Databricks account:

1. Log in to your [Azure Databricks account console](https://accounts.azuredatabricks.net/login) as an account admin.
2. From the left menu of the account console, click **User management**.
3. From the tabs along the top of the _User management_ page, click the **Service principals** tab.
4. In the upper right of the _Service principals_ page, click **Add service principal**.
5. On the _Add service principal_ page, enter a name for the service principal.
6. Under _UUID_, paste the **Application (client) ID** for the service principal.
7. Click **Add**.

### Assign a service principal to a workspace

To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page.

#### Identity federation enabled

To assign a service principal to your Azure Databricks account:

1. Log in to your Databricks [account console](https://accounts.cloud.databricks.com/) as an account admin.
2. From the left menu of the account console, click **Workspaces** and then select a workspace to which you want to add the service principal.
3. From the tabs along the top of your workspace page, click the **Permissions** tab.
4. In the upper right of the _Permissions_ page, click **Add permissions**.
5. In the _Add permissions_ dialog, enter the following details:
 1. For _User, group, or service principal_, select the [service principal](#create-a-service-principal-1) you created.
 2. For _Permission_, click the dropdown to select workspace **User**.

#### Identity federation disabled

To assign a service principal to your Azure Databricks workspace:

1. Log in to your Azure Databricks workspace as a workspace admin.
2. From the top right of your workspace, click your username, and then from the dropdown, click **Admin Settings**.
3. In the left menu of the _Settings_ page, under the _Workspace admin_ subheading, click **Identity and access**.
4. On the _Identity and access_ page, under _Management and permissions_, next to _Service principals_, click **Manage**. 
5. In the upper right of the _Service principals_ page, click **Add service principal**.
6. In the _Add service principal_ dialog, click the **Add new** button.
7. For _New service principal display name_, paste the _Application (client) ID_ for the [service principal](#create-a-service-principal-1), enter a display name, and then click **Add**.

## Grant permissions to crawl metadata

You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan.

To extract metadata, you can grant the [BROWSE privilege](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges.html#browse), currently in public preview. You no longer require the _Data Reader_ preset that granted the following privileges on objects in the catalog - `USE CATALOG`, `USE SCHEMA`, `EXECUTE`, `READ VOLUME`, and `SELECT`.

To grant permissions to a user or service principal:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu of your workspace, click **Catalog**.
3. In the left menu of the _Catalog Explorer_ page, select the catalog you want to crawl in Atlan.
4. From the tabs along the top of your workspace page, click the **Permissions** tab and then click the **Grant** button.
5. In the _Grant on (workspace name)_ dialog, configure the following:
 1. Under _Principals_, click the dropdown and then select the user or service principal.
 2. Under _Privileges_, check the **BROWSE** privilege.
 3. At the bottom of the dialog, click **Grant**.
6. (Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan.

### System tables extraction method

To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse. Follow these steps to extract metadata using system tables:

1. Create one of the following authentication methods:
 - [Personal access token](#personal-access-token-authentication)
 - [AWS service principal](#aws-service-principal-authentication)
 - [Azure service principal](#azure-service-principal-authentication)

2. Grant the following privileges to the identity you created:
 - `CAN_USE` on a SQL warehouse
 - `USE CATALOG` on `system` catalog
 - `USE SCHEMA` on `system.information_schema`
 - `SELECT` on the following tables:
 - `system.information_schema.volumes`
 - `system.information_schema.external_locations`
 - `system.information_schema.catalogs`
 - `system.information_schema.schemata`
 - `system.information_schema.tables`
 - `system.information_schema.columns`
 - `system.information_schema.key_column_usage`
 - `system.information_schema.table_constraints`

### Cross-workspace extraction 

To crawl metadata from all workspaces within a Databricks metastore using a single connection, see [Set up cross-workspace extraction](https://docs.atlan.com/llms/connectors/databricks/set-up-cross-workspace-extraction/llms.txt) for instructions.

## (Optional) Grant permissions to extract primary and foreign keys via system tables

To extract primary key and foreign key relationships for your Databricks tables, you must grant `SELECT` permission. You can grant this permission at the table, schema, or catalog level depending on your requirements:

- **Table level**: Grants access to specific tables only.
- **Schema level**: Grants access to all tables within a schema.
- **Catalog level**: Grants access to all tables within a catalog.

To grant permissions to extract primary and foreign keys:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu of your workspace, click **Catalog**.
3. In the left menu of the _Catalog Explorer_ page, select the catalog, schema, or table for which you want to extract primary and foreign key information.
4. From the tabs along the top, click the **Permissions** tab and then click the **Grant** button.
5. In the _Grant_ dialog, configure the following:
 1. Under _Principals_, click the dropdown and then select the user or service principal.
 2. Under _Privileges_, check the **SELECT** privilege.
 3. At the bottom of the dialog, click **Grant**.
6. Repeat steps 3-5 for each catalog, schema, or table from which you want to extract primary and foreign key relationships.

## (Optional) Grant permissions to crawl external locations

To crawl external locations, you must grant `BROWSE` on individual external locations as below

`GRANT BROWSE
ON EXTERNAL LOCATION 
TO '';`

## (Optional) Grant permissions to query and preview data

:::warning Important!
Atlan currently only supports [querying data](https://docs.atlan.com/llms/catalog/insights/query-data/llms.txt) and [viewing sample data preview](https://docs.atlan.com/llms/catalog/discovery/provide-credentials-to-view-sample-data/llms.txt) for the [personal access token](#personal-access-token-authentication) authentication method.
:::

To grant permissions to query data and preview example data:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu of your workspace, click **Catalog**.
3. In the left menu of the _Catalog Explorer_ page, select the catalog you want to query and preview data from in Atlan.
4. From the tabs along the top of your workspace page, click the **Permissions** tab and then click the **Grant** button.
5. In the _Grant on (workspace name)_ dialog, configure the following:
 1. Under _Principals_, click the dropdown and then select the user or service principal.
 2. Under _Privilege presets_, click the dropdown and then click **Data Reader** to enable read-only access to the catalog. Doing so automatically selects the following privileges - `USE CATALOG`, `USE SCHEMA`, `EXECUTE`, `READ VOLUME`, and `SELECT`.
 3. At the bottom of the dialog, click **Grant**.
6. (Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan.

## (Optional) Grant permissions to import and update tags

To [import Databricks tags](https://docs.atlan.com/llms/connectors/databricks/manage-databricks-tags/llms.txt), you must have a Unity Catalog-enabled workspace and a SQL warehouse configured. Atlan supports importing Databricks tags using system tables for all three authentication methods.

Once you have created a [personal access token](#personal-access-token-authentication), an [AWS service principal](#aws-service-principal-authentication), or an [Azure service principal](#azure-service-principal-authentication), you need to grant the following privileges:

- `CAN_USE` on a SQL warehouse
- `USE CATALOG` on `system catalog`
- `USE SCHEMA`on `system.information_schema`
- `SELECT` on the following tables:
 - `system.information_schema.catalog_tags`
 - `system.information_schema.schema_tags`
 - `system.information_schema.table_tags`
 - `system.information_schema.column_tags`

To push tags updated for assets in Atlan to Databricks, you need to grant the following [privileges](https://docs.databricks.com/en/database-objects/tags.html#requirements):

- `APPLY TAG` on the object
- `USE CATALOG` on the object's parent catalog
- `USE SCHEMA` on the object's parent schema

## (Optional) Grant permissions to extract lineage and usage from system tables

You must have a Unity Catalog-enabled workspace to use system tables.

Atlan supports extracting the following for your Databricks assets using [system tables](https://docs.databricks.com/en/admin/system-tables/index.html#grant-access-to-system-tables):

- [lineage](https://docs.atlan.com/llms/connectors/databricks/extract-lineage-and-usage-from-databricks/llms.txt)
- [usage and popularity metrics](https://docs.atlan.com/llms/connectors/databricks/extract-lineage-and-usage-from-databricks/llms.txt)

### Enable system.access schema

You need your account admin to enable the `system.access` schema using the [SystemSchemas API](https://docs.databricks.com/api/workspace/systemschemas). This enables Atlan to extract lineage using system tables.

In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage.

To verify that system schemas are enabled for each schema, follow the steps in [Databricks documentation](https://docs.databricks.com/api/workspace/systemschemas/list):

- [List system schemas](https://docs.databricks.com/api/workspace/systemschemas/list) using the SystemSchemas API to check the status.
- If enabled for any given schema, the [state](https://docs.databricks.com/en/admin/system-tables/index.html#list-available-system-schemas) is `EnableCompleted`. This confirms that the schema has been enabled for that specific metastore.
- Atlan can only extract lineage using system tables when the state is marked as `EnableCompleted`.

### (Optional) enable `system.information_schema.table`

To generate lineage with the target type set as `PATH` for a table, Atlan uses metadata from `system.information_schema.table` to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables.

#### Grant permissions

:::warning Who can do this?
 You must be a metastore admin, have the `MANAGE` privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions.
:::

In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore from which you want to extract lineage.

1. Open **Catalog Explorer** in your Databricks workspace.
2. Navigate to the catalog (for example, `main`) and then to the appropriate schema (for example, `sales`).
3. Click the **Permissions** tab.
4. Click **Grant**.
5. Enter the user or group name (principal).
6. Assign the following permissions:
 - `USAGE` on the catalog 
 - `USAGE` on the schema 
 - `SELECT` on each relevant table
7. Click **Grant** to apply the changes.

These privileges enable Atlan to read table definitions and other metadata from the metastore.

### (Optional) enable system.query schema

This is only required if you also want to extract [usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) from Databricks.

You need your account admin to enable the `system.query` schema using the [SystemSchemas API](https://docs.databricks.com/api/workspace/systemschemas). This enables Atlan to mine query history using system tables for usage and popularity metrics.

To verify that system schemas is enabled for each schema, follow the steps in [Databricks documentation](https://docs.databricks.com/api/workspace/systemschemas/list). If enabled for any given schema, the [state](https://docs.databricks.com/en/admin/system-tables/index.html#list-available-system-schemas) is `EnableCompleted`.

:::info
💪 **Did you know?** Can't grant `SELECT` permissions on the system tables in `system.access` and `system.query`? Skip the previous steps and create cloned views in a separate catalog and schema. See [Create cloned views of system tables](#optional-create-cloned-views-of-system-tables).
:::

### Grant permissions

Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for [all three authentication methods](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt).

Once you have created a [personal access token](#personal-access-token-authentication), an [AWS service principal](#aws-service-principal-authentication), or an [Azure service principal](#azure-service-principal-authentication), you need to grant the following permissions:

- `CAN_USE` on a SQL warehouse
- `USE_CATALOG` on `system` catalog
- `USE SCHEMA` on `system.access` schema
- `USE SCHEMA` on `system.query` schema (tomine query history for usage and popularity metrics)
- `SELECT` on the following tables:
 - `system.query.history`(to mine query history for usage and popularity metrics)
 - `system.access.table_lineage`
 - `system.access.column_lineage`

You need to [create a Databricks connection in Atlan](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks/llms.txt) for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the _Host_ for the connection.

:::info
💪 **Did you know?** Can't grant `SELECT` permissions on the system tables in `system.access` and `system.query`? Skip the previous steps and create cloned views in a separate catalog and schema. See [Create cloned views of system tables](#optional-create-cloned-views-of-system-tables).
:::

### (Optional) Create cloned views of system tables

When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema.

Follow these steps to set up cloned views:

- Create a catalog and schema to store cloned views. Use meaningful and unique names—for example, `atlan_cloned_catalog` and `atlan_cloned_schema`.
- Create cloned views for the following system tables:

 - **Lineage tables**

 ```sql
 CREATE OR REPLACE VIEW <cloned-catalog-name>.<cloned-schema-name>.column_lineage AS
 SELECT * FROM system.access.column_lineage;

 CREATE OR REPLACE VIEW <cloned-catalog-name>.<cloned-schema-name>.table_lineage AS
 SELECT * FROM system.access.table_lineage;
 ```

 - Replace `<cloned-catalog-name>` and `<cloned-schema-name>` with the catalog and schema names used in your environment.

 - **Popularity metrics**

 ```sql
 CREATE OR REPLACE VIEW <cloned-catalog-name>.<cloned-schema-name>.query_history AS
 SELECT * FROM system.query.history;
 ```

 - Replace `<cloned-catalog-name>` and `<cloned-schema-name>` with the catalog and schema names used in your environment.

#### Grant permissions

Grant the following permissions to enable access to the cloned views:

- `CAN_USE` on a SQL warehouse
- `USE CATALOG` on the catalog (for example, `<cloned-catalog-name>`)
- `USE SCHEMA` and `SELECT` on the schema (for example, `<cloned-catalog-name>.<cloned-schema-name>`)

You must [create a Databricks connection in Atlan](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks/llms.txt) for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the *Host* for the connection.

### Locate warehouse ID

To extract lineage and usage and popularity metrics using system tables, you also need the [warehouse ID of your SQL warehouse](https://docs.atlan.com/llms/connectors/databricks/extract-lineage-and-usage-from-databricks/llms.txt).

To locate the warehouse ID:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu of your workspace, click **SQL Warehouses**.
3. On the _Compute_ page, select the warehouse you want to use.
4. From the _Overview_ tab of your warehouse page, next to the _Name_ of your warehouse, copy the value for your SQL warehouse _ID_. For example, `example-warehouse (ID: 123ab4c5def67890)`, copy the value `123ab4c5def67890` and store it in a secure location.

## (Optional) Grant view permissions to access Databricks entities via APIs

Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets. Use the steps below for each object type to grant **CAN VIEW** permission to the Databricks user or service principal configured in your integration:

1. **Notebook API**(`/api/2.0/workspace/list`): Grant **CAN VIEW** permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see [Manage Access Control Lists with Folders](https://docs.databricks.com/security/access-control/lists.html#manage-access-control-lists-with-folders).

2. **Queries API**(`/api/2.0/sql/queries`): Grant **CAN VIEW** permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see [View Queries](https://docs.databricks.com/sql/users-guide/queries/view-queries.html).

3. **Job API**(`/api/2.2/jobs/list`): Grant **CAN VIEW** permission on each job object directly. 
 Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see [Control Access to a Job](https://docs.databricks.com/jobs/jobs-access-control.html).

4. **Pipeline API**(`/api/2.0/pipelines`): Grant **CAN VIEW** permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see [Configure Pipeline Permissions](https://docs.databricks.com/delta-live-tables/security.html#configure-pipeline-permissions).

## (Optional) Grant permissions for views and materialized views

Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu of your workspace, click **Catalog**.
3. In the _Catalog Explorer_, select the catalog you want to extract view definitions from and generate lineage for in Atlan.
4. From the tabs at the top, click the **Permissions** tab, and then click **Grant**.
5. In the **Grant on (workspace name)** dialog, configure the following:
 - Select the **user** or **service principal** under **Principals**.
 - Select the following privileges under **Privilege presets**:
 - `USE CATALOG`
 - `USE SCHEMA`
 - `SELECT`
6. Click **Grant** to apply the permissions.
7. Repeat steps 3&ndash;6 for each catalog you want to crawl in Atlan.

:::info **Did you know?** 
`SELECT` permission is required to extract the definitions of views and materialized views. If you prefer not to grant `SELECT` at the catalog level, you can grant it on individual views and materialized views instead.
:::

## (Optional) Grant permissions to mine query history

To [mine query history](https://docs.atlan.com/llms/connectors/databricks/extract-lineage-and-usage-from-databricks/llms.txt) using REST API, you need to assign the `CAN MANAGE` permission on your SQL warehouses to the user or service principal.

To grant permissions to mine query history:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu of your workspace, click **SQL Warehouses**.
3. On the _Compute_ page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click **Permissions**.
4. In the _Manage permissions_ dialog, configure the following:
 1. In the _Type to add multiple users or groups_ field, search for and select a user or service principal.
 2. Expand the _Can use_ permissions dropdown and then select **Can manage**. This permission enables the service principal to [view all queries for the warehouse](https://docs.databricks.com/en/security/auth-authz/access-control/index.html#sql-warehouses).
 3. Click **Add** to assign the `CAN MANAGE` permission to the service principal.

### Hive metastore catalog 

The hive_metastore catalog is Databricks' legacy metastore that predates Unity Catalog. If your Databricks workspace uses Unity Catalog, use the [system tables extraction method](#system-tables-extraction-method) instead.

For workspaces still using the hive_metastore catalog, Atlan extracts metadata using the JDBC connection method described below.

#### JDBC metadata extraction method

To crawl metadata from the `hive_metastore` catalog using JDBC, use [personal access token authentication](#personal-access-token-authentication). Service principal authentication isn't supported for this extraction method.

##### Grant permissions

Grant the following privileges to the user who created the personal access token:

- `READ_METADATA` on the `hive_metastore` catalog
- `USAGE` on the `hive_metastore` catalog

:::info
Databricks automatically grants permissions on all child assets (schemas, tables, views, and columns) when you grant permissions on the `hive_metastore` catalog.
:::

To grant these permissions:

1. Log in to your Databricks workspace as a workspace admin.
2. From the left menu, click **Catalog**.
3. In the left menu of the Catalog Explorer page, select the `hive_metastore` catalog.
4. Click the **Permissions** tab, then click **Grant**.
5. In the Grant on hive_metastore dialog, configure the following:
 1. Under Principals, select the user who created the personal access token.
 2. Under Privileges, check both **READ_METADATA** and **USAGE**.
 3. Click **Grant**.

---