What does Atlan crawl from Fivetran?

Did you know?

Fivetran has deprecated its Metadata API and released a more comprehensive Platform Connector. Atlan recommends transitioning to the Fivetran Platform Connector in Fivetran for seamless metadata enrichment in Atlan. For any existing Fivetran workflows, you can migrate to the Fivetran Platform Connector in Fivetran first and then modify the crawler configuration in Atlan to update the extraction method.

Lineage

Atlan uses Fivetran's log events from the Fivetran Platform Connector to generate lineage associated with Fivetran connectors. This is particularly useful for creating lineage between different tools, such as Salesforce and Snowflake.

danger

The assets involved in lineage (tables, columns, objects, fields, and so on) must already be crawled by Atlan before running the Fivetran Enrichment package to enrich them.

Specifically, Atlan will:

• Create lineage between each data asset in Atlan that is associated with a Fivetran connector. (For example, between Salesforce objects and Snowflake tables.)
  • Atlan creates Fivetran Process objects for each data asset that is replicated.
  • Atlan creates column-level lineage to connect the sources (inputs) and destinations (outputs) of each Process. (For example, between Salesforce fields and Snowflake columns.)
• Link each Process to its corresponding connector in the Fivetran UI.
• For any Fivetran sources or destinations that are not natively supported or are supported but have not been crawled yet, Atlan will create partial assets to provide you with a complete view of lineage.

Supported sources and destinations for Fivetran Platform Connector

Sources

Atlan's Fivetran Platform Connector integration supports all sources listed in Fivetran documentation for lineage.

For any sources that are not natively supported or are supported but have not been crawled yet, Atlan will create partial assets to provide you with a complete view of lineage. Note that Atlan will only catalog partial column assets and generate column-level lineage for SQL sources natively supported in Atlan.

Destinations

Atlan's Fivetran Platform Connector integration supports all destinations listed in Fivetran documentation for lineage.

For any destinations that are not natively supported or are supported but have not been crawled yet, Atlan will create partial assets to provide you with a complete view of lineage. Note that Atlan will only catalog partial column assets and generate column-level lineage for SQL sources natively supported in Atlan.

Note that for crawler configuration, only the destinations listed here are supported.

Connectors

Atlan maps the following metadata from Fivetran to its FivetranConnector asset type. This is only applicable to metadata crawled using the Fivetran Platform Connector.

Source property	Atlan property	Where in Atlan
connector.NAME	fivetranConnectorConnectorName	overview sidebar
connector_type.ID	fivetranConnectorConnectorType	overview sidebar
dynamically created by script	fivetranConnectorConnectorURL	overview sidebar
destination.NAME	fivetranConnectorDestinationName	overview sidebar
destination.TYPE	fivetranConnectorDestinationType	overview sidebar
dynamically created by script	fivetranConnectorDestinationURL	overview sidebar
connector.SIGNED_UP	fivetranConnectorSyncSetupOn	overview sidebar
connector.SYNC_FREQUENCY	fivetranConnectorSyncFrequency	overview sidebar
connector.PAUSED	fivetranConnectorSyncPaused	API only
user.GIVEN_NAME + user.FAMILY_NAME	fivetranConnectorSyncSetupUserFullName	API only
connector.USER_EMAIL	fivetranConnectorSyncSetupUserEmail	API only
incremental_mar. INCREMENTAL_ROWS	fivetranConnectorMonthlyActiveRowsFree	overview sidebar
incremental_mar. INCREMENTAL_ROWS	fivetranConnectorMonthlyActiveRowsPaid	overview sidebar
fivetranConnectorMonthlyActiveRowsFree + fivetranConnectorMonthlyActiveRowsPaid	fivetranConnectorMonthlyActiveRowsTotal	API only
calculated by script	fivetranConnectorMonthlyActiveRowsChangePercentageFree	API only
calculated by script	fivetranConnectorMonthlyActiveRowsChangePercentagePaid	API only
calculated by script	fivetranConnectorMonthlyActiveRowsChangePercentageTotal	overview sidebar
calculated by script	fivetranConnectorMonthlyActiveRowsFreePercentageOfAccount	API only
calculated by script	fivetranConnectorMonthlyActiveRowsPaidPercentageOfAccount	overview sidebar
calculated by script	fivetranConnectorMonthlyActiveRowsTotalPercentageOfAccount	API only
dynamically generated by script	sourceURL	overview sidebar
connector_type.CREATED_AT	sourceCreatedAt	asset preview, profile, and filter, overview sidebar
calculated from LOG table	fivetranConnectorLastSyncId	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncStartedAt	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncFinishedAt	overview sidebar
calculated from LOG table	fivetranLastSyncStatusfivetranConnectorLastSyncReason	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncTaskType	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncRescheduledAt	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncTablesSynced	overview sidebar
calculated from LOG table	fivetranLastSyncRecordsUpdated	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncExtractTimeSeconds	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncExtractVolumeMegabytes	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncLoadTimeSeconds	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncLoadVolumeMegabytes	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncProcessTimeSeconds	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncProcessVolumeMegabytes	overview sidebar
calculated from LOG table	fivetranConnectorLastSyncTotalTimeSeconds	overview sidebar