Skip to main content

Troubleshooting Microsoft Azure Data Factory connectivity

Known limitations of Microsoft Azure data factory connector​

Following are the known limitations of the Microsoft Azure Data Factory connector:

  • Lineage isn't supported for data loaded through data flow activity.
  • The ability to include or exclude certain pipelines or activities is currently not supported.
  • Datasets defined in Microsoft Azure Data Factory are currently not mapped to actual data assets in Atlan.
  • Atlan currently doesn't display pipelines as DAGs on the lineage graph. Only the activities that are part of a pipeline are displayed as related assets.
  • Lineage isn't supported for activities that don't have a defined source and sink attribute in their configuration.

Why is lineage missing?​

Remember that the Microsoft Azure Data Factory package only enriches data assets that already exist in Atlan.

Check that the data assets corresponding to Microsoft Azure Data Factory's sources and sinks already exist in Atlan. For example, that the Microsoft Azure Cosmos DB assets have been crawled from Microsoft Azure Cosmos DB, Snowflake assets have been crawled from Snowflake, and more.

The fact that a crawler has run (such as Microsoft Azure Cosmos DB or Snowflake) doesn't mean all assets from that source have been crawled. The filtering configuration of the crawler can omit assets. Or the credentials configured in the crawler may not have access to all of the assets.