Skip to main content

How dbt connectivity works

TL;DR

Learn the three traffic paths between your dbt setup and Atlan - the dbt Cloud API pull, the storage-bucket read, and your own upload pipeline.

Connect

This page explains the three traffic paths between your dbt setup and Atlan and who owns each one. No prior cloud provider knowledge is required.

Pull, not push​

Atlan never receives anything from dbt automatically - it pulls. On a schedule (or when you click Run), an Atlan workflow reaches out and reads your dbt metadata. Where it reaches depends on your dbt flavor:

  • dbt Cloud is like a library with a front desk: Atlan walks up with a membership card (your API token) and asks for the records. If the library also checks IDs at the door (IP restrictions), Atlan's address must be on the guest list too - the card alone isn't enough.
  • dbt Core has no front desk. Instead, every time your pipeline runs dbt, it drops the paperwork (manifest.json and friends) into a filing cabinet (a cloud storage bucket), and Atlan is given a key to read - never write - that cabinet.

Three traffic paths​

Diagram of the dbt connectivity paths from your Atlan tenant: for dbt Cloud, the crawler workflow calls the Admin and Discovery APIs at your access URL with your token (token check, optional IP restrictions); for dbt Core, Atlan's outbound identity (fixed NAT IP + cloud role/ARN) reads manifest.json and run_results.json from a storage bucket (S3 / GCS / ADLS - yours or Atlan-managed), which your own dbt pipeline (CI/CD) uploads the artifacts to

Atlan uses path 1 or path 2, never both for the same connection. Path 3 never touches Atlan - but if it breaks, Atlan sees stale or missing files.

  1. Atlan → dbt Cloud API (dbt Cloud connections). Encrypted HTTPS calls from your tenant's fixed outbound IP to your dbt Cloud access URL. Two things can block it: a token without the right permissions, and dbt Cloud's optional IP-restrictions feature rejecting Atlan's IP.
  2. Atlan → storage bucket (dbt Core connections). Atlan authenticates with your cloud provider (an IAM role on AWS, a service account key on GCP, a service principal on Azure) and reads the dbt output files. Almost every failure here is a permissions or bucket-policy issue.
  3. Your pipeline → the bucket (dbt Core only). Uploading fresh manifest.json files after each dbt run is your job (your CI/CD does it). If uploads stop, the Atlan connection still "works" - it just crawls old metadata.

Glossary​

For definitions of every term used in these guides - API token, service account token vs PAT, access URL, IP restrictions, NAT IP, bucket/prefix, dbt artifacts, IAM role/ARN, trust policy, external ID, KMS key, preflight checks, connection admin - see the Network connectivity glossary.

See also​