Skip to main content
Community Hub

Set up dbt Core

TL;DR

Configure access and storage for dbt Core in Atlan, including credential setup for AWS, GCP, and Azure, and uploading dbt metadata files.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

This guide explains how to set up dbt Core in Atlan, including configuring access, organizing your storage bucket, and uploading the necessary metadata files so Atlan can process and analyze your dbt project data.

Setup and access management​

In this section, learn how to configure access for dbt Core so Atlan can connect to your storage location and read the required metadata. Choose between using your own cloud storage bucket or an Atlan-managed bucket.

Use this option if you store dbt artifacts in your own cloud storage bucket. You create a dedicated read credential for Atlan, then configure the connector in Atlan with your bucket details.

For the IAM role setup (requesting the service identity ARN, creating the IAM policy, and configuring the trust policy), follow Connect dbt Core via a storage bucket—Phase 2. Return here with your Role ARN to complete the Atlan configuration.

Step 1: Configure connector in Atlan​

  1. Go to Marketplace → search for dbt → click to set up dbt.
  2. For Source, select Core.
  3. For Manifest Source, select External Object Storage.
  4. For Storage Provider, select AWS. Set Authentication to IAM Role.
  5. Enter:
    • AWS Role ARN: the Role ARN from Phase 2 of the connect guide
    • Bucket Name: your S3 bucket name
    • Prefix: the path within the bucket where dbt artifacts are stored
    • Region: your bucket's AWS region
  6. Click Test Authentication to verify, then proceed to configure the crawler.

Structure the bucket​

Once you have configured access, the next step is to organize your storage bucket so that Atlan can correctly identify and process uploaded files.

info

Atlan uses the metadata.invocation_id and metadata.project_id attributes to uniquely identify and link the uploaded files. Atlan doesn't use the file paths to identify a project or job that the file belongs to. The following directory structure is provided as a guideline.

Atlan supports extracting dbt metadata from multiple or single dbt projects. The main-prefix has the following format gcs|s3://<BUCKET_NAME>/<PATH_PREFIX> or abfss://<CONTAINER>/<PATH>, if you used Atlan's bucket, the Atlan support team provides it after setting up access policies on your bucket.

note

The <PATH_PREFIX> (or <PATH> for Azure) is optional. If your dbt project directories live at the bucket or container root, leave the Prefix field empty when you configure the crawler and place your project folders directly under the bucket or container.

You need to use the following directory structure, even if you have a single dbt project:

main-prefix
- project1
    - job1
        - manifest.json
        - other files
    - job2
        - manifest.json
        - other files
    - job4
        - manifest.json
        - other files
- project3
    - job5
        - manifest.json
        - other files
warning

Atlan downloads every file under the configured prefix on each workflow run. The include and exclude folder filters are applied after the download and match only the first folder level under the prefix, so they skip processing of excluded folders but don't reduce download time or disk use. If the prefix also contains artifacts from previous runs, workflow runs slow down as the files accumulate and can fail with an out-of-disk-space error.

Upload project files​

To load correct metadata, Atlan processes the manifest.json and run_results.json files for each job. There are many ways to load the metadata, below are suggested approaches from Atlan. You need to upload the files from the target directory of the dbt project into distinct folders. Upload the run artifacts generated from the following commands:

  • (Required) Compilation results:
dbt compile --full-refresh

This command generates files that contain a full representation of your dbt project's resources, including models, tests, macros, node configurations, resource properties, and more.

Files to upload: manifest.json and run_results.json

Alternatively, you can upload the same files by running the dbt run --full-refresh command.

  • (Optional) Test results:
dbt test

This command executes all dbt tests in a dbt project and generates files that contain the test results.

Files to upload: manifest.json and run_results.json

  • (Optional) Catalog:
dbt docs generate

This command generates metadata about the tables and views produced by the models in your dbt project, for example, column data types and table statistics.

Files to upload: manifest.json and catalog.json

Best practices for managing artifacts​

Keep only the latest artifacts for each project and job under the configured prefix. As noted in Structure the bucket, Atlan downloads every file under the prefix on each workflow run. Artifacts from previous runs make each workflow run slower and grow storage use. A job that runs every 15 minutes can add several GB per day if your pipeline writes a new timestamped folder for each run.

Atlan processes every manifest.json, run_results.json, and catalog.json under the prefix, not only the latest ones. dbt Core artifacts have no job identifier, so Atlan groups them by the project_id in each manifest and combines the results for each asset across all runs:

  • For each asset, the result from the run with the most recent manifest generated_at timestamp wins, including its status. For tests, a run that executed the test always takes precedence over a run that only compiled it.
  • For models, if the winning run has no owner, comment, table statistics, columns, or compiled code, Atlan keeps those values from an earlier run.

Re-running the same commands therefore rarely changes what appears in Atlan, but every older copy is still downloaded and parsed on each workflow run.

Recommended: provide only the latest run data for each job. Replace the artifacts after each run by syncing the output of each dbt command over the same project and job folders, so the prefix always holds one current copy of each file.

If you want to provide more data, or keep run history, in the configured prefix, use one of the following approaches:

  • Add a lifecycle (cleanup) policy on the bucket that deletes old run folders after 7 days. Keep a latest copy of the artifacts for each job and exclude that folder from the policy.
  • Write run history to a separate prefix, and point Atlan at a prefix that holds only the latest files.
note

Keep a current copy of the artifacts for every job step, including steps that run rarely or have stopped running, for example a weekly job. Once old run folders age out, Atlan has no artifacts left for those steps and their results disappear from Atlan.