## Extract lineage and usage from Databricks URL: https://docs.atlan.com/apps/connectors/data-warehouses/databricks/how-tos/extract-lineage-and-usage-from-databricks > Retrieve lineage from Unity Catalog and usage and popularity metrics from query history or system tables using REST API, offline, or system table extraction methods. Atlan pulls Databricks lineage from Unity Catalog and usage metrics from system tables, so you see how data moves and how heavily it's used. Retrieve lineage from [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/index.html) and [usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) from system tables. :::warning Databricks query text masking Databricks has been rolling out masking of SQL query text since **June 22, 2026**, and it becomes the default for all accounts on **August 26, 2026**. This affects **usage and popularity metrics** and **SQL-level lineage detail** for Databricks assets in Atlan. Table-level and column-level lineage from Unity Catalog system tables aren't affected. For full details on what changes and what stays the same, see [Query text access in Databricks](https://docs.atlan.com/llms/connectors/databricks/databricks-sql-masking-change/llms.txt). ::: :::danger Deprecated The **REST API** extraction method for lineage and popularity is **deprecated**. Use the **system tables** method instead, which provides more reliable and comprehensive extraction. The REST API extraction method is scheduled for removal in a future release. ::: ## Prerequisites Before you begin, make sure you have: * [Set up authentication](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt) using personal access token, AWS service principal, or Azure service principal * [Crawled assets from Databricks](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks/llms.txt) * Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) to understand workflow sequencing * Unity Catalog enabled on your Databricks workspace for lineage and usage metrics extraction. You may also need to [upgrade existing tables and views to Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/migrate.html) and reach out to your Databricks account executive to enable lineage in Unity Catalog. ## Create extraction workflow To extract lineage and usage from Databricks: 1. In your Atlan workspace, click **Connectors** in the left sidebar. * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation. 2. Click **Marketplace**. 3. Search for **Databricks Miner**. The Marketplace lists two tiles by Atlan with this name; select the one described as **Package to extract lineage information and usage metrics from Databricks**. 4. Click **Install**. 5. Once installation completes, click **Setup Workflow** on the same tile. ### Configure lineage extraction :::warning Offline extraction sunset The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt). ::: ### REST API (Deprecated) :::danger Deprecated The REST API lineage extraction method is **deprecated**. Use the [system tables](#configure-lineage-extraction) method instead. ::: Atlan connects to your database and extracts lineage directly. 1. For **Connection**, select the connection to extract. The [crawler](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks/llms.txt) must have already run for you to select a connection. 2. Click **Next**. ### System Table Atlan connects to your database and [queries system tables](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt) to extract lineage directly. 1. For **Connection**, select the connection to extract. The [crawler](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks/llms.txt) must have already run for you to select a connection. 2. For **Extraction Catalog Type**, choose one of the following: * **Default**: Select to fetch lineage from the system catalog and `access` schema. * **Cloned_catalog**: Select to fetch lineage from a cloned catalog and schema. Before proceeding, make sure the following prerequisites are met: * You have already created cloned views named `column_lineage` and `table_lineage` in your schema. If not, follow the steps in [Create cloned views of system tables](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt). * The `atlan-user` must have `SELECT` permissions on both views to access lineage data. Then, provide values for the following fields: * **Cloned Catalog Name**: Catalog containing the cloned views. * **Cloned Schema Name**: Schema containing the cloned views. 3. For **SQL Warehouse ID**, enter the [ID you copied from your SQL warehouse](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt). 4. If you want to enable lineage tracking at the file path level for volumes or external locations, enable **File Path Lineage**. 5. If you want to build lineage from [Databricks AI/BI dashboards](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks-ai-bi-dashboards/llms.txt) to the tables and columns they read, enable **Enable Dashboard Lineage**. This option appears only when the private preview is enabled for your tenant, and requires dashboards to have been crawled first. 6. Click **Next**. ### Configure usage extraction :::warning Important! Query history is currently in public preview for Databricks. ::: Atlan extracts [usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) from [query history](https://docs.databricks.com/api/workspace/queryhistory/list) or [system tables](https://docs.databricks.com/en/admin/system-tables/index.html#grant-access-to-system-tables). This feature is currently limited to queries on SQL warehouses; queries on interactive clusters aren't supported. Additionally, expensive queries and compute costs for Databricks assets are currently unavailable due to limitations of the [Databricks APIs](https://docs.databricks.com/api/workspace/queryhistory/list). 1. For **Fetch Query History and Calculate Popularity**, click **Yes** to retrieve [usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) for your Databricks assets. To skip this step, click **No** and proceed to [Run extractor](#run-extractor). 2. For **Popularity Extraction Method**, choose one of the following: * **REST API** *(Deprecated)*: Extract usage and popularity metrics from query history. This method is deprecated—use system tables instead. * **System table** *(Recommended)*: Extract metrics directly from system tables. Then configure: * **Extraction catalog type for popularity**: Choose where to fetch popularity data from: * **Default**: Uses the system catalog and `query` schema to fetch popularity metrics. * **Cloned_catalog**: Select to fetch popularity from cloned views in a separate catalog and schema. Before proceeding: * The `query_history` view must exist in the provided schema. * The `atlan-user` must have `SELECT` permission on the view. Then provide: * **Cloned Catalog Name**: The catalog that contains the `query_history` view. * **Cloned Schema Name**: The schema that contains the `query_history` view. For more information, see [Create cloned views of system tables](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt). * **SQL Warehouse ID**: Enter the [ID you copied from your SQL warehouse](https://docs.atlan.com/llms/connectors/databricks/set-up-databricks/llms.txt). #### Configure usage settings 1. For **Popularity Window (days)**, enter the number of days to include. The maximum limit is 30 days. 2. For **Start time**, choose the earliest date from which to mine query history. If you're using the offline extraction method to extract query history from Databricks, skip this field. If running the miner for the first time, Atlan recommends setting a start date around three days prior to the current date and then scheduling it daily to build up to two weeks of query history. Mining two weeks of query history on the first miner run may cause delays. For all subsequent runs, Atlan requires a minimum lag of 24 to 48 hours to capture all the relevant transformations that were part of a session. Learn more about the miner logic at [Troubleshooting usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/troubleshooting-usage-and-popularity-metrics/llms.txt). 3. For **Excluded Users**, type the names of users to be excluded while calculating [usage metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) for Databricks assets. Press `enter` after each name to add more names. ### Run extractor 1. To check for any [permissions or other configuration issues](https://docs.atlan.com/llms/connectors/databricks/preflight-checks-for-databricks/llms.txt) before running the extractor, click **Preflight checks**. This is currently only supported when using REST API and offline extraction methods. If you're using system tables, skip to step 2. 2. Choose how to run the extractor: * To run the extractor once immediately, at the bottom of the screen, click **Run**. * To schedule the extractor to run hourly, daily, weekly, or monthly, at the bottom of the screen, click **Schedule Run**. Once the extractor completes, you can view lineage for Databricks assets. ## See also * [Extract on-premises Databricks lineage](https://docs.atlan.com/llms/connectors/databricks/extract-on-premises-databricks-lineage/llms.txt): Extract lineage from on-premises Databricks instances using the databricks-extractor tool and upload results to S3. * [What does Atlan crawl from Databricks](https://docs.atlan.com/llms/connectors/databricks/what-does-atlan-crawl-from-databricks/llms.txt): Reference documentation for all metadata assets, attributes, and relationships extracted by the Databricks connector. ---