Bring dbt Core metadata into Atlan from dbt Projects on Snowflake. The Snowflake crawler copies each run's dbt files, and the dbt workflow reads them. No bucket or dbt Cloud needed.
Connect dbt Core via dbt Projects on Snowflake Private Preview
Reading dbt Core metadata from dbt Projects on Snowflake is in private preview. To enable it for your workspace or share feedback, raise a support ticket or reach out to your Atlan customer success manager.
Snowflake can store and run a dbt project itself. You deploy the project as a DBT PROJECT object and run it with EXECUTE DBT PROJECT, usually from a Snowflake task. For each run, Snowflake keeps the files that dbt writes.
Atlan uses three of these files. It reads them only from each run's dbt_artifacts.zip archive. Atlan finds the archive with the Snowflake function SYSTEM$LOCATE_DBT_ARTIFACTS and downloads it with GET.
| File | What Atlan gets from it |
|---|---|
manifest.json | dbt models, sources, tests, seeds, lineage, descriptions, tags, and the columns that your dbt project documents |
run_results.json | Model statuses and test results |
catalog.json (optional) | Every column of each model, source, and seed, with its data type. See Write catalog.json for column metadata. |
How it works
- Your dbt project runs in Snowflake. Snowflake keeps the files of each run.
- Each Snowflake crawl copies the files of the runs from the last 7 days into Atlan's storage, up to 30 runs for each project. Each crawl that copies at least one run makes a new copy.
- The dbt workflow reads the newest copy for the selected Snowflake connection. If a crawl copies no files, the previous copy stays the newest.
You don't need a storage bucket or dbt Cloud. You only need the Snowflake connection.
Prerequisites
- A Snowflake connection in Atlan. See Crawl Snowflake.
- Admin or connection admin privileges in Atlan.
- Someone who can grant privileges in Snowflake.
Prepare your dbt projects in Snowflake
Deploy and run each dbt project
Deploy the dbt project as a dbt project object. Use Deploy in Snowsight, or CREATE DBT PROJECT. Run it with EXECUTE DBT PROJECT <database>.<schema>.<project>.
Atlan reads only the runs of a deployed dbt project object. Atlan doesn't read runs that you start from a Snowsight workspace without deploying the project.
Grant access to Atlan's role
Atlan's role is the role in the credentials of your Snowflake connection in Atlan. Grant this role MONITOR on each dbt project:
GRANT MONITOR ON DBT PROJECT <database>.<schema>.<project> TO ROLE <atlan_role>;
To also cover dbt projects that you create later in a schema, add a future grant:
GRANT MONITOR ON FUTURE DBT PROJECTS IN SCHEMA <database>.<schema> TO ROLE <atlan_role>;
A future grant covers only the dbt projects created after it. Grant each existing project with the first statement.
MONITOR is the minimum privilege. Both extraction methods of the Snowflake crawler, Information Schema and Account Usage, need this grant. See also Snowflake permissions.
Include each project in your Snowflake crawl
The include and exclude filters of the Snowflake crawler control which dbt projects Atlan reads. The crawler reads a dbt project only if the filter includes the project's database and schema.
The filter works at the database or schema level. For example, if two dbt projects are in the schema ANALYTICS.DBT, Atlan reads both or neither. If you exclude that schema, the Snowflake crawler also skips the tables in it.
Run each project at least once every 7 days
Each Snowflake crawl copies the runs of the last 7 days. Atlan copies only the runs with the state SUCCESS or HANDLED_ERROR. A dbt project with no such run in the last 7 days isn't included in that crawl's copy.
Know which files Atlan uses
Atlan uses manifest.json, run_results.json, and optionally catalog.json from each run. Atlan supports these files from both dbt Core 1.x and dbt Fusion 2.x.
If several runs include the same model or test, Atlan uses the newest result.
Write catalog.json for column metadata
catalog.json is optional. Column data types and undocumented columns come from it. Add a run that writes it after your build or run, for example as a second Snowflake task. The command depends on the dbt version of the run:
| dbt version | Command |
|---|---|
| dbt Core 1.x | docs generate |
| dbt Fusion 2.0.0 | compile --write-catalog |
For dbt Core 1.x:
EXECUTE DBT PROJECT <database>.<schema>.<project> ARGS = 'docs generate --target <target>';
For dbt Fusion 2.0.0:
EXECUTE DBT PROJECT <database>.<schema>.<project> ARGS = 'compile --write-catalog --target <target>';
On dbt Fusion 2.0.0, build --write-catalog doesn't write catalog.json. dbt fixed this in 2.0.8. See the dbt v2.0.8 release notes.
Without catalog.json, Atlan still creates the dbt assets and lineage. Atlan then shows only the columns that your dbt project documents, without data types.
Configure Atlan
- Run or schedule the Snowflake crawler for the connection. The Information Schema and Account Usage extraction methods both work.
- Create the dbt workflow. See Crawl dbt for the full steps.
- For Source, select Core.
- For Manifest Source, select Snowflake Connection.
- In the Snowflake Connection field, select the Snowflake connection.
- In the Connection field of the dbt configuration step, select the same Snowflake connection. If you leave it empty, dbt assets can link to matching tables in every Snowflake connection. For example, if a production and a development connection both crawl
ANALYTICS.DBT.ORDERS, the dbt modelorderscan link to both tables. - Schedule the dbt workflow to run after the Snowflake crawler. The dbt workflow reads the newest copy. If the dbt workflow runs first, it reads the copy from the previous crawl.
The Include Folder Filter and Exclude Folder Filter fields don't show for this manifest source. The Snowflake crawler filter chooses the dbt projects.
If Atlan finds no dbt files, the preflight check fails and suggests this action:
Run the Snowflake crawler for the Snowflake connection selected under Manifest Source, with the dbt project permissions listed in the docs, then re-run the dbt workflow.
Limitations
- This feature is in private preview.
- Atlan doesn't read runs started from a Snowsight workspace without a deployed dbt project.
- You select dbt projects per database or schema, through the Snowflake crawler filter.
- The dbt workflow reads only the files that the last Snowflake crawl copied. Run the Snowflake crawler first, so that it copies the newest dbt project files. Then run the dbt workflow, so that the dbt app processes them.
See also
- Choose how Atlan connects to dbt: Compare the ways Atlan reads dbt metadata.
- Crawl dbt: Configure and run the dbt workflow.
- Crawl Snowflake: Configure the Snowflake crawler and its include and exclude filter.