Databricks connectivity

||Connect docs via MCP

Connectivity and network issues

Symptom-first: find what you're seeing in the left column, ordered by how often each issue actually occurs. Each entry includes a check you can run yourself before opening a ticket.

Quick index

You see	Most likely cause	Jump to
Source IP address <ip> is blocked by Databricks IP ACL for workspace <id>	Stale or missing entry in the workspace IP access list	IP access list
Could not resolve host: adb-….azuredatabricks.net or "Name or service not known," but the URL opens fine in your browser	Workspace is behind a private endpoint; no DNS path from Atlan	DNS resolution
Connection times out; the private endpoint was recently created	Endpoint pending approval (Azure) or VPCE not registered (AWS)	Pending endpoint
Endpoint shows Approved / Succeeded, but port 443 still times out	NSG missing Atlan's CIDR, or destination-side drop	Approved but dead
Test connection passes but SQL warehouses don't list	API path only partially reachable, or DNS enforced differently for the warehouse path	Warehouses missing
Worked for months, broke overnight, nothing changed on your side	Egress IP change or access-list edit	Sudden breakage
MCP / OAuth from Databricks fails: "Client not found," tenant-selection page, or connection timed out	Shared-proxy endpoints or missing inbound allowlist	Special cases: MCP

Blocked by Databricks IP ACL

Symptom. Workflows fail with:

Source IP address: 203.0.113.28 is blocked by Databricks IP ACL
for workspace: 3833858263208367

Why. The workspace's IP access list doesn't include the address Atlan is currently arriving from. Either the list was created without Atlan's IPs, or Atlan's published egress IPs changed (announced 30 days ahead, check the network details page).

Fix. The error message hands you the answer: the IP in the error is Atlan's current source. Compare it against the published values for your tenant's region, add the full current set to the access list, and remove retired entries. Over Private Link, remember the list must contain the private CIDR ranges instead.

DNS resolution failures from Atlan only

Symptom. Crawls fail with curl: (6) Could not resolve host or UnknownHostException, yet the same workspace URL opens fine from your laptop.

Why. "Resolves for me" and "resolves for Atlan" are different networks. The usual causes, in order:

1. The workspace is integrated with a private endpoint and public DNS resolution is disabled, Atlan needs its own private endpoint and DNS mapping to reach it. Start the AWS / Azure guide.
2. A Private Link was set up, but the DNS mapping covers a different hostname, a second URL, a renamed workspace, or a new workspace that was never added. Confirm the exact FQDN on your support ticket.
3. The hostname was mistyped in the connection config, compare character-by-character.

Check yourself. Ask whoever owns the workspace: "is this workspace behind a private endpoint / is public network access disabled?" That one answer routes the whole investigation.

Endpoint stuck in pending or not registered

Symptom. Timeouts on a newly created private link.

• Azure: open the workspace resource → Networking → Private endpoint connections. If Atlan's endpoint shows Pending, approve it, nothing approves itself. If it's absent entirely, Phase 2 hasn't completed; check the ticket.
• AWS: there is no pending request in your AWS console, acceptance happens by registering the VPCE ID in the Databricks Account Console and attaching it to the workspace's network configuration. If you were waiting for an AWS-side approval prompt, this is the misunderstanding.

Approved, but port 443 is dead

Symptom. The endpoint shows Approved / Succeeded, DNS resolves to the right private IP, and TCP 443 still times out.

Why. Approval is control-plane; traffic is data-plane. In observed cases the packet leaves Atlan correctly and is dropped at or beyond the endpoint. Checklist, in order:

1. NSG rules on the private endpoint's subnet, do they permit Atlan's private CIDR (from your ticket) inbound on TCP 443? This is the most common miss.
2. Workspace-side network controls, any ingress filtering, firewall, or routing policy your platform team applies to the Databricks path.
3. If both check out, ask Atlan support to share its network-path evidence (DNS resolution, NSG diagnostics, probe results) and set up a joint call with Databricks support and your cloud provider. Destination-side Private Link drops have required Microsoft involvement to resolve, escalate with evidence rather than re-checking the approved state for days.

Test connection passes, SQL warehouses don't list

Symptom. Credentials validate, but the crawler setup can't list SQL warehouses, or the workflow fails at the warehouse-listing step with a DNS or timeout error.

1. Confirm the workspace URL used by the warehouse path is the same one the DNS mapping covers, and that warehouses were declared in scope when the Private Link was set up.
2. Ask whether the workspace enforces private-only DNS, publicly browsable from your office network but unresolvable from Atlan means the private path is required (see DNS resolution).
3. Confirm the service principal or PAT actually has CAN USE on at least one running SQL warehouse, an empty list is sometimes permissions, not network.

It worked for months, then broke overnight

1. Check the network details page, did a published Atlan value change? (Atlan announces 30 days ahead by email to tenant admins; check spam/distribution lists.)
2. Ask your Databricks admin whether the IP access list was created or edited, security rollouts that forget Atlan's IPs are the single most common cause of overnight breakage.
3. Ask your platform team whether a workspace migration, rename, or network change happened, a new workspace URL needs a new DNS mapping (see Special cases).
4. Grab the exact error from the failed workflow run, if it names a blocked source IP, IP access list has your answer in one step.

Opening ticket that gets solved fast

If you still need support, include these six things, they usually turn a multi-day exchange into one reply:

• Your Atlan tenant URL and the failing workflow run link
• Connectivity method (public / AWS PrivateLink / Azure Private Link / VPN)
• The workspace URL and workspace ID (and Resource ID on Azure), private connectivity is per workspace, so "the Databricks workspace" isn't enough
• The exact error text from the workflow logs
• Whether the workspace enforces an IP access list, front-end Private Link, or public-access-disabled, ask your Databricks admin if unsure
• Whether it ever worked, and if so when it stopped

Setup, authentication, and metadata

Questions about authentication, permissions, extraction methods, and the metadata Atlan does or doesn't capture. Find your question below.

Quick index

Topic	Question	Jump to
Cost and popularity	Are expensive queries or compute costs considered?	Compute costs
Cost and popularity	How's popularity calculated?	Popularity
Authentication	Test authentication or preflight check errors	Auth and preflight errors
Performance	Extraction slower than usual	Slow extraction
Metadata	Some metadata is missing	Missing metadata
Querying	SQL doesn't work when querying	SQL support
Compute	Using Atlan when the compute engine is off	Compute engine off
Assets and permissions	Not all assets appear in Atlan	Missing assets
Authentication	Test authentication is very slow	Slow auth
Extraction	REST API (Unity Catalog) limitations	REST API limits
Authentication	Worked before, now failing (expired PAT)	Workflow now failing
Unity Catalog	Migrating to Unity Catalog	UC migration
Metadata	Notebooks missing from extraction	Missing notebooks
Metadata	Metadata missing for some entities	Cross-workspace entities
Metadata	Nested columns beyond level 30	Nested columns
Multi-workspace	Service principal loses access to a workspace	Workspace access lost
Assets and permissions	Catalog assets not appearing	Catalog assets missing
Lineage	Lineage view incomplete	Incomplete lineage

Does Atlan consider expensive queries and compute costs?

No, Atlan doesn't factor in expensive queries or compute costs due to limitations in the Databricks APIs, which don't expose this information.

How does Atlan calculate popularity for Databricks assets?

Atlan calculates popularity for tables, views, and columns in Databricks by analyzing query execution data. It retrieves query history from the system.query.history table and specifically filters for execution_status = 'FINISHED' and statement_type = 'SELECT' to determine how frequently assets are accessed.

How to debug test authentication and preflight check errors?

Hostname resolution error

Provided Host name cannot be resolved via DNS, please check and try again.

• The hostname you have provided can't be resolved through DNS. Check that the hostname is correct.
• Verify that the DNS settings have been configured properly.

Invalid client ID or secret

Provided Client ID is invalid, please check and try again.

• The client ID or secret you have provided is either invalid or no longer working. Follow the steps for AWS or Azure setup to generate new credentials.

Invalid tenant ID

Provided tenant ID is invalid, please check and try again.

• The tenant ID you have provided is incorrect.
• Make sure the tenant ID you have provided corresponds to the one in your Microsoft Entra ID application.

Unity Catalog not linked

Configured Databricks instance doesn't have Unity Catalog linked. Please choose JDBC extraction instead of REST API in Atlan.

• If you haven't set up Unity Catalog in your Databricks workspace, you can change the extraction method to JDBC instead of REST API to crawl your Databricks assets in Atlan.

Connection timeout

Failed to connect to Databricks (connection timed out). Please check your host and port and try again.

• The connection to the Databricks instance has timed out.
• Verify that the host and port are correct.
• Check that no firewall rules or network issues are blocking the connection.

Invalid HTTP path

Provided HTTP path is invalid, please check and try again.

• The HTTP path you have provided is invalid.
• Make sure the endpoint is properly configured and accessible, and the warehouse ID in the HTTP path is correct.

Invalid personal access token

PAT token is invalid, please check and try again.

• The personal access token used for authentication is invalid.
• Make sure the token is valid and neither deleted nor expired.
• You can also generate a new personal access token, if needed.

Insufficient permissions for crawling metadata

User doesn't have access to any schemas / dbs, please check the accesses provided to the atlan user and try again.

• Check that the service principal or the user who's PAT token is being used has the necessary permissions provided. Refer to the setup doc to understand permissions required for different auth types.

Insufficient permissions for some of the included crawling metadata

Warning, user doesn't have access to the following objects anymore, or the objects no longer exist on the source!, check failed for ...

• user doesn't have access to one or more db objects from the include filter, (such as catalogs / schemas).
• You can either remove these objects from the include filter if they no longer exist on the source.
• Or check that the service principal or the user who's PAT token is being used has the necessary permissions provided. Refer to the setup doc to understand permissions required for different auth types.

Insufficient permissions to crawl tags

User doesn't have access to the following system tables

• Check that you have sufficient permissions provided for the tags extraction.

User doesn't have permission to access warehouses

please check your credentials and warehouse access

• Check that the configured user / service principal has CAN_USE on the configured SQL warehouse.

Unable to access query history from the source, user doesn't have the access

• Check the permissions required for the system tables based lineage extraction are provided.

System table extraction checks failing with

User doesn't have access to the following system tables

• Check the permissions required for the system tables based extraction.

General connection failure

Unable to connect to the configured Databricks instance, please check your credentials and configs and then try again. If the problem persists, contact support@atlan.com.

• Check that you have entered the host and port correctly.
• Verify that the credentials for the connection are correct.
• Check that your Databricks instance is properly configured and available.
• If the problem still persists after verifying all of the previous steps, contact Atlan support.

Why does workflow take longer than usual in extraction?

• Certain Databricks runtime versions don't have an easy way to extract some metadata (for example partitioning, table_type, and format). Extra operations must be performed to retrieve these, resulting in slower performance.
• If you aren't already, you may want to try the Unity Catalog extraction method.

Why is some metadata missing?

• When using incremental extraction, consider running a one-time full extraction to capture any newly introduced metadata.

• Currently, some metadata can't be extracted from Databricks:

  Metadata	JDBC	REST API	System Tables
  ViewCount and TableCount (on schemas)	❌	✅	✅
  RowCount (on tables and views)	❌	❌	❌
  TABLE_KIND (on tables and views)	❌	❌	❌
  PARTITION_STRATEGY (on tables and views)	❌	❌	❌
  CONSTRAINT_TYPE (on columns)	❌	✅	✅
  Partition key (on columns)	❌	✅	✅
  Table partitioning information	✅	❌	✅
  BYTES, SIZEINBYTES (table size)	❌	❌	❌

• The team is exploring ways to bring this metadata into Atlan if Databricks supports extraction of the metadata.

Why doesn't my SQL work when querying Databricks?

• Atlan currently supports SparkSQL on Databricks runtime 7.x and above.

Can I use Atlan when Databricks compute engine isn't running?

• Atlan needs the Databricks compute engine to be running for two activities:
  • Crawling assets (normal and scheduled run)
  • Querying assets (including data previews)
• If you don't need to perform the activities listed, your experience shouldn't be affected.
• In any other case, you'll get a downgraded experience on Atlan if the compute engine isn't running. Queries won't work as expected and a scheduled workflow might fail after a couple of retries.
• The team recommends turning off the Terminate after x minutes of inactivity option in your cluster to avoid these problems. If you have this turned on, any of the listed activities triggers the cluster to come back online within about 30 seconds.

Why can't I see all assets on Atlan that are available in Databricks?

• Have you excluded the database or schema when crawling?
• Does the Databricks user you configured for crawling have access to these other assets?

Why is test authentication taking so long?

• Please check the state of the compute engine. It must be in a running state for all operations, including authentication.

What limitations are there with REST API (Unity Catalog) extraction method?

• Currently, schema-level filtering and retrieving table partitioning information aren't supported.

Why has my workflow started to fail when it worked before?

• This can happen if the PAT you configured the workflow with has since expired.
• You need to create a new PAT in Databricks, and then modify the workflow configuration in Atlan with this new PAT.
• If you are unable to update the PAT, pause the workflow and reach out to us.

How do I migrate to Unity Catalog?

• Currently Unity Catalog is in a public preview state.
• The Databricks team is working on an automated migration to Unity Catalog.
• Currently you must migrate individual tables manually.

Why are some notebooks missing from metadata extraction?

Notebooks stored inside hidden directories (names starting with "." such as .hidden_dir/) are generally not returned by the /api/2.0/workspace/list API endpoint. This may cause missing notebook details in Atlan.

Why is metadata missing for some Databricks entities?

The Databricks APIs used provide data only within a single configured workspace. If an entity used in lineage creation exists outside this workspace, its details won't be available via these APIs.

Does Atlan support nested columns beyond level 30?

Atlan doesn't support nested columns beyond 30 levels for complex types such as Struct, Array, and Map. Columns exceeding this nesting depth aren't parsed. Instead, the deepest column level gets assigned the data type string, and its value contains a string representation of the remaining nested structure.

For example, LEVEL_31 has the data type <LEVEL_32:STRUCT<LEVEL_33:STRUCT<...>>>.

What happens if service principal loses access to one workspace?

The crawler is resilient to this scenario. During the discovery phase, it fails to connect to that specific workspace, logs it as inaccessible, and simply skips it. The process continues for all other available workspaces without failing the entire run.

Why are assets from specific catalog not appearing in Atlan?

This is almost always a permission issue. Verify that the service principal has been granted USE CATALOG, BROWSE, and SELECT permissions on the catalog and its contents (schemas, tables).

Why is my lineage view incomplete?

Check the source and target tables of the missing lineage link. The service principal must have SELECT permissions on both tables for lineage to be captured.

For more information on cross-workspace extraction setup, see Set up cross-workspace extraction.