## Crawl dbt

URL: https://docs.atlan.com/apps/connectors/etl-tools/dbt/how-tos/crawl-dbt

> Configure and run the crawler to extract metadata from your dbt Cloud or dbt Core projects.

Atlan crawls dbt Cloud or dbt Core to enrich assets with model information, lineage, and documentation.

## Prerequisites

Before you begin, make sure you have:
* Completed [dbt Cloud setup](https://docs.atlan.com/llms/connectors/dbt/set-up-dbt-cloud/llms.txt) (for dbt Cloud) or [dbt Core setup](https://docs.atlan.com/llms/connectors/dbt/set-up-dbt-core/llms.txt) (for dbt Core)
* Admin or connection admin privileges in Atlan
* Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows

## Create crawler workflow

Follow these steps to create a workflow in Atlan that captures metadata from dbt.

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **dbt Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Configure authentication

Choose your dbt source and provide the required credentials.

### dbt Cloud

1. For **Extraction method**, click **Cloud**.

2. For **Host Name**, enter the domain name of your dbt Cloud instance, if not the default. Include the `https://`. For example:
	```
	https://cloud.getdbt.com
	```
	For more information on access URLs, refer to [dbt documentation](https://docs.getdbt.com/docs/cloud/about-cloud/access-regions-ip-addresses).
3. For **Authentication Type**, select **Service Account** or **PAT** depending on your token type.

4. Enter your dbt Cloud token in the **Token** field.

5. Click **Test Authentication** to verify the connection.

### dbt Core

1. For **Extraction method**, click **Object storage**.

2. Select your Cloud Provider, AWS, GCP, or Azure.

3. Choose the Authentication type (for example, IAM Role) and provide the required role or access details.

4. Enter your **Bucket name** and **Region**. You can find these details in your cloud storage configuration.

5. (Optional) Enter a **Prefix** to point Atlan at a subfolder that holds your dbt project directories. Leave this field empty to crawl from the bucket or container root.

6. Click **Test Authentication** to validate access.

### Configure connection

To complete the dbt connection configuration:

1. Provide a **Connection Name** that represents your source environment. For example, you might use values like `analytics`, `production`, or `development`.

2. (Optional) To change the users able to manage this connection, change the users or groups listed under **Connection Admins**.

 :::warning

 If you don't specify any user or group, nobody can manage the connection - not even admins.

 :::

3. At the bottom of the screen, click **Next** to proceed.

### Configure dbt settings

The configuration options change based on the **Extraction method** you selected earlier, Cloud or Core (object storage). Follow this step to fine-tune how dbt metadata is enriched in Atlan.

### dbt Cloud

1. Under **Exclude Metadata**, choose projects or environments you don't want to include in enrichment. Leave blank if you want all available projects.

2. Under **Include Metadata**, select specific projects or environments to include.

3. To limit the enrichment to a particular connection with materialized assets, click **Connection** and select the relevant option. (This defaults to all connections, if none are specified.)

4. For **Import Tags**, click **Yes** to sync dbt tags from your Cloud workspace into Atlan.

5. For **Enrich Metadata in Materialized Assets**, click **Yes** to enable enrichment for both dbt and materialized assets.

### dbt Core

1. To limit the enrichment to a particular connection with materialized assets, click **Connection** and select the relevant option. (This defaults to all connections, if none are specified.)

2. If you want to import tags defined in your dbt files, click **Yes** under **Import Tags**.

### Run crawler

To run the dbt crawler, after completing the previous steps:

1. To check for any [permissions or other configuration issues](https://docs.atlan.com/llms/connectors/dbt/preflight-checks-for-dbt/llms.txt) before running the crawler, click **Preflight checks**.

2. You can either:
	* To run the crawler once immediately, at the bottom of the screen, click the **Run** button.
	* To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule Run** button.

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

## See also

* [Set up dbt Cloud](https://docs.atlan.com/llms/connectors/dbt/set-up-dbt-cloud/llms.txt): Configure authentication for dbt Cloud
* [Set up dbt Core](https://docs.atlan.com/llms/connectors/dbt/set-up-dbt-core/llms.txt): Upload dbt Core project files to cloud storage
* [Preflight checks for dbt](https://docs.atlan.com/llms/connectors/dbt/preflight-checks-for-dbt/llms.txt): Verify permissions and configuration before crawling
* [Manage dbt tags](https://docs.atlan.com/llms/connectors/dbt/manage-dbt-tags/llms.txt): Import and manage tags from dbt in Atlan

---