Crawl Fivetran

✦TL;DR

Enrich Atlan with Fivetran metadata to track data pipeline lineage and transformations. Configure the Fivetran Platform Connector and connect a supported data warehouse destination.

||

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

Atlan crawls Fivetran to catalog data pipelines and transformations and to build source-to-destination lineage.

Establish a connection between Atlan and your Fivetran instance to enrich your workspace with metadata about data pipelines and transformations. This enables you to track lineage, understand data movements, and govern your ETL processes. The Fivetran Platform Connector must be configured and running in your Fivetran environment. Review the order of operations for metadata enrichment workflows before starting.

Select the source

To select Fivetran as your source:

1. In your Atlan workspace, click Connectors in the left sidebar.

   • If you are using the Old UI (Classic), click New Workflow in the top navigation.

2. Click Marketplace.

3. Search for Fivetran Enrichment and select it.

4. Click Install.

5. Once installation completes, click Setup Workflow on the same tile.

Provide credentials

In order to run this package, you must ensure the following:

• The Fivetran Platform Connector is set up and has run successfully in Fivetran at least once.
• Fivetran logs are stored in a destination supported by Atlan.
• The above destination has been crawled in Atlan.

To use the Fivetran Platform Connector:

1. For Atlan Connection, Atlan will use the credentials of your selected connection to read the Fivetran Platform Connector tables associated with that connection. You can either:

   • Create a connection in Atlan for the destination warehouse you configured while setting up the Fivetran Platform Connector in Fivetran. This connection in Atlan must have access to the Fivetran tables created by the Fivetran Platform Connector. Atlan supports the following destinations:

     • Amazon Redshift
     • Databricks
     • Google BigQuery
     • PostgreSQL
     • Snowflake

   • If you have already created a connection in Atlan, select the connection to extract. (To select a connection, the crawler must have already run for a supported destination.)

     warning

     If you have an existing connection, you must ensure that the user or other access permissions configured for that connection allow access to the Fivetran tables created by the Fivetran Platform Connector or update them accordingly.

2. For Fivetran Platform Schema, select the destination schema where Fivetran logs are stored. You can only select one schema and must ensure that the connection above has access to all Fivetran log tables stored in this destination schema.

3. (Optional) To show where upstream data comes from, enter a Fivetran API key and secret. Refer to Add Fivetran API key.

4. Once successful, at the bottom of the screen, click Next.

Add Fivetran API key (optional)

Without a Fivetran API key, Atlan shows the host and the database of database sources when the Fivetran log tables include them. With a key, Atlan also reads each Fivetran connection from the Fivetran API.

What the key adds

• For file sources, the folder URL and the folder prefix.
• For database sources, the host and the database when the Fivetran log tables don't include them.

Atlan shows these values on the lineage hover, in the overview of the upstream (partial) asset, and in the Fivetran connector drawer.

Enter the key

On the Fivetran credential form, enter the values in these two optional fields:

• Fivetran API key (optional)
• Fivetran API secret (optional)

When you edit the credential, type the API secret again. If you leave it empty, the stored secret is removed, and the next crawl keeps the last source locations and logs a warning.

Remove the key

To stop using the key, clear both the API key and the API secret. The next crawl removes the values that only the key gave, and keeps the values from the Fivetran log tables. If you clear only one of the two fields, the next crawl keeps the last source locations and logs a warning.

Required access

Use a Fivetran API key that can read the connections in scope.

SharePoint limits

For SharePoint, Atlan shows the folder URL only for links that contain the folder path. Sharing links of the form /:f:/s/... give the folder prefix only, not the folder URL. For example, https://contoso.sharepoint.com/:f:/s/... gives the prefix only.

If a Fivetran lookup fails

• Atlan keeps the values stored from the previous crawl.
• If Atlan can't read the key state and can't read the stored values either, the crawl stops before publish. The catalog keeps its values.

Configure the connection

To complete the Fivetran connection configuration:

1. Provide a Connection Name that represents your source environment. For example, you might use values like production, development, gold, or analytics.

2. (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins.

   warning

   If you do not specify any user or group, nobody will be able to manage the connection - not even admins.

Run the enrichment

You can now enrich Atlan with Fivetran metadata:

1. To check for any permissions or other configuration issues before running the enrichment, click Preflight checks.

2. You can either:

   • To run the crawler once immediately, at the bottom of the screen, click the Run button.
   • To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule & Run button.

Once the enrichment has completed running, you will see lineage extended upstream from your data platform, warehouse, or lake! 🎉