Skip to main content
Community Hub

What lineage does Atlan extract from Microsoft Fabric

TL;DR

Learn about the lineage relationships that Atlan can map from Microsoft Fabric. The Microsoft Fabric connector is generally available, and Atlan is available in the Microsoft Marketplace and is Microsoft Azure co-sell eligible.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

Atlan extracts lineage from Microsoft Fabric in two modes depending on whether Scanner API Access is enabled in the crawler workflow. The mode affects both the depth of lineage and which assets are part of the lineage graph.

Lineage by mode​

Lineage capabilityScanner API enabledScanner API disabled
Lakehouse/Warehouse Table → Semantic Model Table✅ Available✅ Available
Lakehouse/Warehouse Column → Semantic Model Table Column✅ Available✅ Available
Upstream lineage (external SQL sources → Lakehouse/Warehouse)✅ Available✅ Available
Lakehouse/Warehouse table or column → Lakehouse shortcut table or column ³✅ Available✅ Available
Source table or column → Mirrored database table or column ⁴✅ Available✅ Available
Mirrored database table or column → Direct Lake semantic model table or column✅ Available✅ Available
Semantic Model Tables → Semantic Model✅ Available✅ Available
Semantic Model → Report✅ Available✅ Available
Dataflow entity column → Semantic Model Table Column✅ Available✅ Available
Source table / column → Dataflow entity column✅ Available✅ Available
Dataflow Gen2 (CI/CD) → destination lakehouse or warehouse table / column ²✅ Available✅ Available
Semantic Model Table Column → Semantic Model Measure✅ Available✅ Available
Semantic Model Measure → Report Page / Visual❌ Not available✅ Available
Semantic Model Table / Column → Report Page❌ Not available✅ Available
Semantic Model Table Column → Report Visual❌ Not available✅ Available
Pipeline Copy Activity (Table → Table)❌ Not available✅ Available
Notebook Spark job: table → table ¹✅ Available✅ Available
Notebook Spark job: column → column ¹✅ Available✅ Available

¹ Requires Extract Spark runtime lineage in the crawler and Spark runtime lineage setup in Fabric. See Spark runtime lineage.

² Requires the Contributor role on each workspace that contains Dataflow Gen2 (CI/CD) items, in both modes. See What Atlan extracts by workspace role.

³ For OneLake shortcuts to Fabric tables. See Shortcut lineage.

⁴ For supported mirroring sources. Requires the Contributor role on the workspace that contains the mirrored database, in both modes. See Mirrored database lineage.

Dataflow lineage

Lineage doesn't yet follow references between queries in the same dataflow, or queries combined with Table.Combine.

For Dataflow Gen2 (CI/CD), Atlan builds lineage from the dataflow definition. If the crawl identity has less than the Contributor role on the workspace, Atlan catalogs the dataflow but extracts no entity columns or lineage for it. This applies when Scanner API Access is enabled too, because no admin API returns Dataflow Gen2 (CI/CD) definitions.

Scanner API enabled​

When Scanner API Access is enabled, Atlan uses the Power BI Admin Scanner APIs to derive lineage. The following lineage paths are available:

  • External sources → Semantic Models: Upstream lineage from Lakehouse and Warehouse tables and columns to their corresponding Semantic Model Tables and Semantic Model Table Columns, including column-level lineage.
  • Upstream SQL lineage: Table- and column-level lineage from external SQL warehouse assets to Fabric assets.

Report Pages, Report Visuals, and Pipeline Copy Activities aren't part of the lineage graph in this mode because those assets aren't cataloged when Scanner API Access is enabled.

Scanner API disabled​

When Scanner API Access is disabled, Atlan uses both admin and non-admin APIs, enabling the full lineage graph:

  • All lineage paths available in scanner mode (following)
  • Semantic Model → Report → Page: Full downstream lineage from Semantic Model Tables and Columns through to Report Pages and Visuals
  • Pipeline Copy Activities: Table-level lineage for Copy Activity source and sink tables within Data Pipelines

Shortcut lineage​

A OneLake shortcut makes a table from another location appear in a lakehouse, without copying the data. Teams often use shortcuts to share data, for example when business teams build on tables from a central lakehouse in another workspace.

Atlan catalogs a shortcut table like any other lakehouse table. When the shortcut points to a table in a Fabric lakehouse or warehouse, Atlan also draws lineage from that source table to the shortcut table, including column-level lineage. This works for shortcuts within a workspace and across workspaces, in both modes.

For this lineage to appear:

  • Both workspaces are crawled: the workspace that contains the shortcut and the workspace that contains its source table are included in the crawl.
  • The crawl identity has Contributor on the shortcut's workspace: Fabric only shows where a shortcut points to identities with the Contributor role. With a lower role, Atlan catalogs the shortcut table without lineage to its source. See Assign workspace permissions.

Atlan catalogs the shortcut table but doesn't draw lineage to its source for these shortcuts:

  • Shortcuts to storage outside Fabric, such as Azure Data Lake Storage, Amazon S3, or Google Cloud Storage
  • Shortcuts that bring in a whole schema instead of a single table
  • Shortcuts to Fabric SQL databases, KQL databases, or mirrored databases

Shortcuts in a lakehouse's Files folder aren't cataloged, because Atlan doesn't catalog files.

Mirrored database lineage​

A mirrored database continuously replicates tables from an external database, such as Snowflake or Azure SQL Database, into OneLake. Fabric doesn't record this replication as lineage. Atlan builds the lineage from what Fabric reports about each mirrored database: its source type, its source database, and the source schema and table of each mirrored table.

For the mirrored database assets and properties that Atlan catalogs, see What does Atlan crawl from Microsoft Fabric.

Upstream lineage from source tables​

For each mirrored table, Atlan draws lineage from the source table to the mirrored table, and from each source column to the mirrored column with the same name. Columns that Fabric adds during replication, such as __sequence_number, __timestamp, METADATA$ROW_ID, or MSSQL_System_Uniquifier_*, have no upstream lineage.

Mirroring sourceAtlan connector that catalogs the sourceMatched onUpstream lineage
SnowflakeSnowflakeDatabase, schema, and table✅ Table and column
Azure SQL DatabaseMicrosoft SQL ServerDatabase, schema, and table✅ Table and column
Azure SQL Managed InstanceMicrosoft SQL ServerDatabase, schema, and table✅ Table and column
SQL ServerMicrosoft SQL ServerDatabase, schema, and table✅ Table and column
Azure Database for PostgreSQLPostgreSQLDatabase, schema, and table✅ Table and column
MySQLMySQLDatabase, schema, and table✅ Table and column
OracleOracleDatabase, schema, and table✅ Table and column
Azure Databricks (Unity Catalog)DatabricksCatalog, schema, and table✅ Table and column
Google BigQueryGoogle BigQueryDataset and table ⁵✅ Table and column
SAP Datasphere——❌ Cataloged only
Open mirroring——❌ Cataloged only
Azure Cosmos DB——❌ Cataloged only

⁵ Fabric doesn't report the Google Cloud project of a BigQuery source. If several projects cataloged in Atlan contain a dataset and table with the same names, the mirrored table can link to the table in another project.

For sources marked Cataloged only, Atlan catalogs the mirrored database, its tables, and its columns, without upstream lineage.

How Atlan matches source tables​

Atlan looks for the source table among assets of the matching connector type, by database, schema, and table name. It doesn't use the source host or server name. If more than one connection of that connector type contains a table with the same names, Atlan links the mirrored table to the first match, which might not be the one you expect.

If the source table isn't cataloged in Atlan, Atlan creates it as a partial asset, so the lineage still appears on the lineage graph. To link the mirrored table to the cataloged source table instead, crawl the source with its Atlan connector, then run the Microsoft Fabric crawler again. See order of operations.

Requirements​

  • Contributor role on the mirrored database's workspace: Atlan reads the source type, source database, and source table names through workspace-level Fabric APIs. With a lower role, Atlan catalogs the mirrored database but might not show its Mirroring source or draw upstream lineage. See What Atlan extracts by workspace role.
  • A workspace role at all: if the crawl identity has no role on the workspace and only the Scanner APIs return the mirrored database, Atlan catalogs the mirrored database without tables, columns, or upstream lineage.

Downstream lineage to semantic models​

Atlan draws table- and column-level lineage from mirrored database tables to the Direct Lake semantic models built on them, in both modes. This works for semantic models in Direct Lake on OneLake mode and Direct Lake on SQL mode.

Query history lineage​

The crawler's preview Mine query history option also covers the SQL analytics endpoints of mirrored databases. When it's turned on, Atlan builds table and column lineage from queries run on those endpoints, as it does for lakehouses and warehouses. This needs a workspace role on each workspace.

Not supported​

Atlan doesn't catalog these Fabric items, so they have no lineage:

  • Fabric SQL databases
  • Cosmos DB in Fabric
  • KQL databases

Spark runtime lineage Private Preview​

When Extract Spark runtime lineage is enabled, Atlan builds lineage from the OpenLineage events that Fabric Spark emits while notebooks run. This lineage covers transformations in notebook code, which metadata extraction alone can't see.

How Atlan models runtime lineage​

  • Notebook: Each notebook in a crawled workspace becomes a FlowControlOperation asset of type Notebook.
  • Spark job: Each distinct Spark job that a notebook runs becomes a FlowControlOperation asset of type Spark Job, linked to its notebook.
  • Table-level lineage: A Process links the job's input tables to its output tables and is linked to the Spark job that orchestrates it.
  • Column-level lineage: A ColumnProcess for each output column links it to the input columns it depends on. Each column process belongs to its job's table-level process.

For the properties Atlan maps on notebooks and Spark jobs, see What does Atlan crawl from Microsoft Fabric.

Atlan links a job's input and output datasets to assets that are already cataloged in Atlan:

  • Lakehouse tables must be cataloged by the same Microsoft Fabric connection. Atlan matches them by the workspace and Lakehouse IDs in the event, and column names must match the cataloged columns exactly, including case.
  • Tables cataloged by another connection must be crawled before the Microsoft Fabric crawler runs. See order of operations.

Datasets that don't match a cataloged asset, such as files under a Lakehouse's Files folder, are skipped. Atlan doesn't create placeholder assets for them. A job gets a table-level process only when at least one input and one output match, and a column process only when its output column and at least one input column match. A Spark job with no matching datasets still appears under its notebook, without lineage.

Column-level dependencies​

Column-level lineage includes input columns that feed an output column's value, whether copied, transformed, or aggregated, and input columns used in a conditional expression that produces the value, such as a CASE WHEN condition. Columns used only to filter, join, group, sort, or window rows aren't linked to output columns.

Which run Atlan shows​

Atlan keeps the latest execution of each Spark job, chosen by event time, and tracks each job independently:

  • When a job runs again, its new execution replaces the earlier lineage.
  • When a later notebook run doesn't include a job, that job keeps the lineage of its last execution.
  • Spark jobs are identified by their job name within the notebook. Write operations that produce the same Spark job name share one Spark job asset, so only the latest of them is kept.

When a notebook is deleted in Fabric or falls outside the crawl's workspace filters, Atlan stops extracting the notebook, its Spark jobs, and their lineage.

When events appear in Atlan​

Fabric stores events in hourly UTC folders, and Atlan reads an hour only after it ends. For example, a crawl at 14:35 UTC reads events up to 14:00 UTC; events from 14:00–14:59 UTC are read by the next crawl after 15:00 UTC.

  • The first crawl of a connection reads events from the previous 30 days.
  • Each later crawl continues from where the previous crawl stopped, and Atlan keeps the lineage it already extracted between crawls.
  • Event files that arrive late for an hour that Atlan already read aren't picked up. To read the previous 30 days again, create a new connection.

See also​