Connect dbt Core via storage bucket

✦TL;DR

Connect dbt Core to Atlan - your pipeline uploads dbt's output files to a cloud bucket (S3, GCS, ADLS, or Atlan-managed) and Atlan reads them.

||

|Connect

dbt Core has no API for Atlan to call - instead your pipeline uploads dbt's output files to a cloud bucket and Atlan reads them. Five phases; the access phase differs by cloud provider.

Prerequisites

• A pipeline (CI/CD, Airflow, cron…) that runs dbt and can upload files afterward.
• Someone who can create IAM roles / service accounts in your cloud account - usually your cloud or platform team.
• Admin or connection admin privileges in Atlan.

Five phases at glance

Phase	What happens	Owner	Typical time
1. Choose storage	Your own bucket (S3/GCS/ADLS) or an Atlan-managed one	Atlan User + Cloud team	One decision
2. Grant access	Give Atlan read access to the bucket	Cloud team + Atlan support	Hours–days (approvals)
3. Upload artifacts	Pipeline writes dbt files in the expected layout	dbt / data team	1–2 hours
4. Configure & preflight	Enter bucket details in Atlan; run preflight (S3)	Atlan User	30 minutes
5. Run & verify	Crawl, check assets, schedule	Atlan User	30 minutes

Phase 1 - Choose your storage

Owner: Atlan User + Your cloud team

	Your own bucket	Atlan-managed bucket
Good when	You already have a bucket and an infra team comfortable with cross-account access	You want zero IAM work, or approvals on your side are slow
Supported clouds	AWS S3, Google Cloud Storage, Azure Data Lake Storage	Provisioned by Atlan (raise a support request)
What Atlan needs	Read access (Phase 2)	Nothing - you get a prefix + upload credentials from support
Preflight checks	S3 only	Available (S3-backed)

Details for both options: Set up dbt Core. If you pick the Atlan-managed bucket, raise the support request now, skip Phase 2, and resume at Phase 3 when the prefix and credentials arrive.

Phase 2 - Grant Atlan read access

Owner: Your cloud team

AWS S3 - assume-role handshake

1. Atlan User - Ask Atlan support for the Atlan dbt service identity ARN for your tenant (it's tenant-specific).
2. Cloud team - Create an IAM policy granting read on the bucket and prefix:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AtlanDbtReadAccess",
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::example-dbt-artifacts",
      "arn:aws:s3:::example-dbt-artifacts/dbt/*"
    ]
  }]
}

3. Cloud team - Create an IAM role with that policy attached, and a trust policy naming Atlan's identity:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "<atlan-dbt-service-identity-arn>" },
    "Action": "sts:AssumeRole"
  }]
}

Two AWS traps seen repeatedly

External ID: if your security standard requires an sts:ExternalId condition in trust policies, confirm current behavior with Atlan support first - the connector hasn't always sent an external ID, so a trust policy requiring one can make AssumeRole fail.

KMS encryption: if the bucket uses a customer-managed KMS key, also grant the role kms:Decrypt and kms:DescribeKey on that key - otherwise reads fail with AccessDenied even though the S3 policy is perfect.

Google Cloud Storage

1. Create a service account and grant it roles/storage.objectViewer on the bucket itself (project-level grants on the wrong project are a recurring miss).
2. Generate a JSON key for it. You'll paste the key contents into Atlan in Phase 4, with the Project ID, bucket name, and prefix.

Azure Data Lake Storage

1. Register an app in Azure AD; note the Application (client) ID and Directory (tenant) ID, and create a client secret.
2. Assign the service principal the Storage Blob Data Reader role on the storage account/container.
3. If the storage account has a firewall, see Allowlists and firewalls - Azure has a same-region quirk that makes IP allowlisting ineffective.

Phase 3 - Upload dbt artifacts

Owner: dbt / data team

After each dbt run, your pipeline uploads the artifacts in this layout (one folder per project, one per job):

<main-prefix>/
├── project1/
│   ├── job1/
│   │   ├── manifest.json      # required - from dbt compile/run --full-refresh
│   │   └── run_results.json   # recommended - test outcomes
│   └── job2/
│       └── manifest.json
└── project2/
    └── job3/
        ├── manifest.json
        └── catalog.json       # optional - from dbt docs generate (column details)

• Atlan identifies files by the metadata.invocation_id and metadata.project_id inside them - folder names are for your sanity, not Atlan's.
• Upload fresh files every run. Stale files = stale catalog, with no error anywhere.
• Keep names exactly manifest.json / run_results.json / catalog.json - renamed or missing manifests are a real failure mode.

Phase 4 - Configure in Atlan and preflight

Owner: Atlan User

1. Open New workflow → dbt Assets, choose Core, then your cloud provider.
2. Bucket name - the plain name only, for example, example-dbt-artifacts. Never include s3:// - the prefix-included form is a known silent breaker.
3. Region - the bucket's region (for example, us-east-1).
4. Prefix - the parent folder for your project folders; leave empty if projects sit at the bucket root.
5. Credentials: AWS - the role ARN your team created in Phase 2; GCS - project ID + service-account JSON; ADLS - tenant ID, client ID, client secret, storage account, container.
6. S3 only: click Preflight checks. It runs the "Manifest file check on S3" - confirming Atlan can list the bucket and see a manifest at your prefix. On GCS/ADLS there is no preflight; your first crawl is the test.
7. Set Connection Admins (empty = nobody can manage it later).

Phase 5 - Run and verify

Owner: Atlan User

Success checklist

• Preflight passes (S3) or the first crawl completes (GCS/ADLS)
• dbt models appear in Atlan with columns and tests
• A second crawl after your next dbt run picks up changes (proves the upload pipeline works end-to-end)
• The crawl is scheduled to run after your dbt jobs finish, not during

Anything failing → Troubleshooting dbt connectivity, which is organized by the exact error text you'll see.

Next steps

Crawl dbt: Configure and run the crawler to extract metadata from dbt