Set up AWS private network link to Databricks
Connect Atlan to an AWS-hosted Databricks workspace over AWS PrivateLink, so traffic never crosses the public internet. Five phases, each labeled with who performs it.
Read How private connectivity works first (5 minutes). It explains every term used below, such as endpoint service, VPCE ID, Private Access Settings, and DNS mapping, and the registration handshake this guide walks through.
Prerequisites
- Databricks Enterprise pricing tier, hosted on AWS, in an E2-supported region. (Databricks on Azure? Use the Azure guide.)
- Your Atlan tenant hosted on AWS. The workspace doesn't need to be in the same region as your tenant, Atlan supports cross-region private connectivity. Different cloud entirely? See Special cases.
- Front-end PrivateLink enabled on the workspace, with a Private Access Settings (PAS) object attached. Your Databricks account admin does this; it's a workspace-update operation.
- A Databricks account admin available for Phases 1 and 3, and an AWS-savvy colleague for sanity checks.
Five phases at glance
Who does what: Databricks admin · Atlan support · Your cloud/network team · You, in Atlan
| Phase | What happens | Owner | Typical time |
|---|---|---|---|
| 1. Gather | Collect workspace and endpoint-service details | Databricks admin | 15 minutes |
| 2. Build | Atlan creates the interface VPC endpoint on its side | Atlan support | 1-2 business days |
| 3. Register | Register Atlan's VPCE in the Databricks Account Console | Databricks admin | 20 minutes |
| 4. Map DNS and allowlist | Atlan maps the workspace URL to the endpoint; you update the IP access list if used | Atlan support + Databricks admin | Same day |
| 5. Connect and verify | Point the Atlan connection at the workspace URL and test | You, in Atlan | 30 minutes |
Phase 1: Gather your workspace details (Databricks admin)
Raise a support request from within Atlan with your Atlan tenant URL and, for each workspace you want connected:
- AWS region of the workspace
- Workspace URL, for example
acme-prod-ws1.cloud.databricks.com - Workspace ID, the number after
?o=in the URL - VPC endpoint service name for the workspace's front end, in the form
com.amazonaws.vpce.<region>.vpce-svc-xxxxxxxxxxxxxxxxx, from the Databricks Account Console under Cloud resources → Network - Front-end PrivateLink status, is it already enabled, and what's the workspace's PAS ID? If not enabled yet, confirm your team plans to enable it
- Whether SQL warehouses are in scope, if Atlan needs to mine query history or run preflight over a SQL warehouse, say so; warehouses ride the same workspace front end, but Atlan verifies them explicitly
Copy-paste request for your admin: "Please send me the workspace's region, URL, workspace ID, and the front-end VPC endpoint service name (Account Console → Cloud resources → Network), plus whether front-end PrivateLink / Private Access Settings is enabled. It contains no secrets, only names and IDs."
Phase 2: Atlan builds its endpoint (Atlan support)
Atlan creates an interface VPC endpoint in your tenant's network targeting your workspace's endpoint service. When done, typically within 1-2 business days, support replies with:
- The VPCE ID you'll register in Phase 3, for example
vpce-0a1b2c3d4e5f6a7b8 - The private CIDR ranges the connection originates from (needed if the workspace enforces an IP access list, see Phase 4)
If multiple workspaces share the same regional endpoint service, they can share one Atlan-side endpoint. Each workspace still needs its own registration (Phase 3) and its own DNS mapping (Phase 4). List every workspace in the ticket so nothing is missed.
Phase 3: Register endpoint in Databricks (Databricks admin)
Databricks won't accept private connections from an unregistered endpoint. In the Databricks Account Console:
- Go to Cloud resources → Network → VPC endpoints and click Register VPC endpoint. Enter the VPCE ID from Phase 2 and the region.
- Attach the registered endpoint to the workspace's network configuration, and confirm the workspace's Private Access Settings lists it (or is set to permit all registered endpoints in the account).
- Decide the PAS public access setting deliberately: leaving public access enabled during migration avoids locking out existing tools; disabling it enforces private-only.
Unlike Azure, nothing shows up as a pending request in your AWS account. The acceptance happens by registering the VPCE ID in the Databricks Account Console, not in the AWS console. If you're looking for a pending connection in AWS and can't find one, this is why.
Phase 4: DNS mapping and allowlist (Atlan support + Databricks admin)
Once you confirm registration on the ticket, Atlan creates a private hosted zone in its network so the workspace URL resolves to the private endpoint. Two things from your side:
- Confirm the exact workspace URLs you'll use in Atlan, the DNS mapping is per hostname. If your team uses more than one URL for the same workspace, name them all.
- If the workspace enforces an IP access list, it must permit the private path. Add Atlan's private CIDR ranges from Phase 2, the public egress IPs are irrelevant over PrivateLink.
Atlan then validates reachability (DNS resolution and TCP 443) from the tenant and confirms on the ticket.
Phase 5: Connect and verify (You)
- In Atlan, open your Databricks connection setup (New workflow → Databricks Assets).
- In the Host field, enter the normal workspace URL, for example
acme-prod-ws1.cloud.databricks.com. Unlike some other connectors, you don't enter a special private hostname; the DNS mapping inside Atlan's network routes it privately. - Enter credentials as usual (service principal recommended) and click Test connection.
Success checklist
- Test connection passes in the Atlan UI
- SQL warehouses are listed in the crawler setup (first proof that the API path works end-to-end)
- Preflight checks pass when you configure the crawler
- A crawl scoped to one small catalog/schema completes
- A full crawl (or miner run) completes
All five green? You're done. Any step failing, see Troubleshooting connectivity.