Set up on-premises Databricks lineage extraction
The Docker-based databricks-extractor offline tool has been sunset and is no longer available. For on-premises or network-restricted Databricks lineage extraction, use one of the supported approaches below.
In some cases you won't be able to expose your Databricks instance for Atlan to extract and ingest lineage. For example, this may happen when security requirements restrict access to sensitive, mission-critical data.
Supported approaches
For Databricks instances that you can't expose to Atlan directly, use one of the following supported approaches to extract lineage:
- Agent extraction with Self-Deployed Runtime - Atlan's agent executes lineage extraction within your own environment. See Self-Deployed Runtime and the Agent extraction method section of the Databricks crawler guide.
- Secure Agent - Fetch lineage from Databricks through Atlan's Secure Agent, running inside your network. See How to configure Secure Agent for workflow execution.
- Direct connectivity via private link - Expose Databricks to Atlan over a private link:
Once connectivity is in place, see Extract lineage and usage from Databricks for the supported lineage extraction methods.
If you have an existing setup that relies on the sunset offline extractor, contact your Atlan Account team to plan your migration.