Databricks
Integrate, catalog, and govern Databricks assets in Atlan.
Atlan crawls Databricks metadata—Unity Catalog schemas, tables, views, volumes, and AI models—and builds column-level lineage automatically.
Overview: Catalog Databricks workspaces, databases, schemas, and tables in Atlan. Gain visibility into lineage, usage, and governance for your Databricks assets.
Get started
Start by setting up the Databricks connector in Atlan. This involves configuring authentication and connection settings so Atlan can access your Databricks workspace.
- How Atlan connects to Databricks: Understand the three connectivity paths—public, Private Link, and site-to-site VPN—and what determines which one applies to your workspace.
- Set up the connector: Configure the Databricks connector with authentication credentials and connection settings.
- Additional configurations needed for your environment:
- Enable SSO for Databricks: Set up SSO authentication if your organization uses single sign-on for Databricks access.
- Set up cross-workspace extraction: Configure a single service principal to crawl metadata from all workspaces within a Databricks metastore, useful when you have multiple workspaces sharing the same metastore.
- Additional configurations needed for your environment:
Crawl assets
After setting up the connector, crawl your Databricks assets to discover and catalog them in Atlan:
- Crawl Databricks assets: Discover and catalog your Databricks assets in Atlan.
AI models
Catalog AI models registered in your Unity Catalog Model Registry and build lineage connecting them to their upstream data dependencies.
- Crawl Databricks AI models: Discover and catalog AI models and their versions from the Unity Catalog Model Registry.
- Extract Databricks AI model lineage: Build upstream lineage between model versions and the datasets, tables, and functions they depend on.
Advanced setup
Use these guides for specialized deployment scenarios or additional configuration options. These guides help you when your Databricks environment requires specific network configurations, runs on-premises, when you need to extract lineage and usage metrics, or when you need to manage tags and descriptions.
Lineage and usage
Extract lineage and usage metrics to understand how data flows through your Databricks assets and which assets are most frequently accessed:
- Extract lineage and usage from Databricks: Extract lineage and usage metrics from your Databricks assets.
On-premises
- Set up on-premises Databricks access: Configure Atlan to access on-premises Databricks environments.
- Crawl on-premises Databricks: Crawl metadata from on-premises Databricks environments.
- Set up on-premises Databricks lineage extraction: Prepare for offline lineage extraction from on-premises Databricks.
- Extract on-premises Databricks lineage: Step-by-step instructions for extracting lineage from on-premises Databricks.
Private network
Set up private network connections to Databricks when you need secure, private connectivity without exposing your Databricks workspace to the public internet:
- Allowlist Atlan IPs in Databricks: Add Atlan's addresses to your Databricks workspace IP access list or a corporate firewall.
- Set up an AWS private network link to Databricks: Establish a secure, private network connection to Databricks on AWS.
- Set up an Azure private network link to Databricks: Establish a secure, private network connection to Databricks on Azure.
- Special cases: Multiple workspaces, extra hostnames, cross-cloud setups, site-to-site VPN, and lakehouse and MCP traffic.
Tag and description management
Configure and manage tags in Databricks to enhance metadata governance, and push the metadata you enrich in Atlan back to Databricks:
- Manage Databricks tags: Configure and manage tags in Databricks.
- Push description updates to Databricks: Enable reverse sync so asset descriptions written in Atlan land in Databricks as comments.
Concepts
- How private connectivity works: What a private link is, who builds which half, and the three traffic paths between Atlan and Databricks.
- Lineage filtering approaches: Understand how Atlan filters Databricks lineage data to show only valid lineage relationships.
References
- What does Atlan crawl from Databricks: Learn about the Databricks assets and metadata that Atlan discovers and catalogs.
- Preflight checks for Databricks: Verify prerequisites before setting up the Databricks connector.
- Image and versioning: Identify the Databricks app container image, its versioning scheme, and how to find the exact version assigned to your tenant.
Troubleshooting
- Databricks connectivity: Resolve common Databricks connection issues and errors.
- Cross-workspace extraction issues: Troubleshoot common issues in Databricks cross-workspace extraction with error, cause, and solution guidance.
FAQ
- Cross-workspace extraction setup: Frequently asked questions about setting up and configuring cross-workspace extraction.
- Atlan vs Databricks lineage: Frequently asked questions about how Atlan lineage differs from Databricks native lineage.
- SQL query text masking change: How Databricks's default SQL query text masking (effective August 26, 2026) affects usage metrics, SQL-level lineage, and SQL Intelligence in Atlan.
- Description reverse sync delays: Why a description edit takes a few minutes to reach Databricks, and what to do when the first edit after enabling reverse sync doesn't arrive.