Skip to main content
Community Hub

Crawl Microsoft Fabric

TL;DR

Crawl metadata from Microsoft Fabric to catalog workspaces, reports, dashboards, and datasets. Extract lineage and analytics information from your Fabric environment. The Microsoft Fabric connector is generally available, and Atlan is available in the Microsoft Marketplace and is Microsoft Azure co-sell eligible.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

Discover and catalog Microsoft Fabric workspaces, reports, dashboards, and datasets in Atlan. After completing the prerequisite setup, you can extract metadata about your analytics and business intelligence assets. Review the order of operations for metadata enrichment workflows before starting.

Prerequisites​

Before you begin, make sure you have:

Create crawler workflow​

To crawl metadata from Microsoft Fabric, review the order of operations and then complete the following steps.

  1. In your Atlan workspace, click Connectors in the left sidebar.

    • If you are using the Old UI (Classic), click New Workflow in the top navigation.

  2. Click Marketplace.

  3. Search for Fabric Assets and select it.

  4. Click Install.

  5. Once installation completes, click Setup Workflow on the same tile.

Configure authentication​

Atlan offers two extraction methods to connect to Microsoft Fabric. Select the method that best fits your security and infrastructure requirements.

Atlan connects directly to Microsoft Fabric to crawl metadata.

  1. For Authentication, select Service Principal.

  2. Enter the service principal credentials you configured when setting up Microsoft Fabric:

    • Tenant ID
    • Client ID
    • Client Secret

  3. (Optional) Toggle Enable Scanner API Access to control how metadata is extracted. See Scanner API Access for details on capabilities and limitations.

  4. Click Test Authentication to confirm connectivity to Microsoft Fabric.

  5. Once successful, click Next.

Scanner API access​

The Enable Scanner API Access toggle controls how Atlan extracts metadata from Microsoft Fabric. The two modes differ in catalog coverage and lineage depth.

CapabilityScanner API enabledScanner API disabled
Catalog Workspaces, Lakehouses, Warehouses, Semantic Models, Reports, Dashboards, Dataflows✅ Available✅ Available
End-to-end lineage (external sources → Semantic Models)✅ Available✅ Available
Catalog Report Pages❌ Not available✅ Available
Catalog Report Visuals❌ Not available✅ Available
End-to-end lineage to Report Pages and Visuals❌ Not available✅ Available
Pipeline Copy Activities cataloged❌ Not available✅ Available
Last updated date and creator for Dataflow Gen1 and Dataflow Gen2❌ Not available✅ Available
Notebooks, Spark jobs, and their lineage (with Extract Spark runtime lineage enabled)✅ Available✅ Available

When to enable Scanner API Access:

Enable the toggle when your service principal or APIM managed identity doesn't have Viewer access to individual workspaces. Scanner API mode uses only the Power BI Admin Scanner APIs (POST /admin/workspaces/getInfo) and doesn't require workspace-level permissions, unless you also enable Extract Spark runtime lineage. This is a common choice for organizations that centralize Fabric administration and prefer not to grant per-workspace access to the crawl identity.

When to disable Scanner API Access:

Disable the toggle when your service principal or APIM managed identity has Viewer access to each workspace. Non-scanner mode uses both scanner and non-scanner APIs, providing full catalog coverage including Report Pages, Visuals, and Pipeline Copy Activities.

For the required permissions for each mode, see Set up Microsoft Fabric.

Configure connection​

  1. Provide a Connection Name that represents your source environment. For example, you might want to use values like production, development, gold, or analytics.

  2. (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins. If you don't specify any user or group, nobody can manage the connection, including admins.

  3. Click Next to proceed.

Configure crawler​

Configure the Microsoft Fabric crawler to specify which workspaces to include or exclude. If a workspace appears in both the include and exclude filters, the exclude filter takes precedence.

  • Include Workspaces: Select Microsoft Fabric workspaces to include. Defaults to all workspaces when left blank.
  • Exclude Workspaces: Select workspaces to exclude. No workspaces are excluded by default.
  • Extract Spark runtime lineage: Turn on to catalog notebooks and their Spark jobs, with table- and column-level lineage from Fabric Spark runtime lineage events. Off by default. Requires Set up Spark runtime lineage, and applies in both Scanner API modes. The service principal needs a workspace role on every included workspace, so use the workspace filters to limit the crawl to workspaces it can access.

Run crawler​

  1. To check for any permissions or configuration issues before running the crawler, click Preflight checks.

  2. You can either:

    • To run the crawler once immediately, click Run.
    • To schedule the crawler to run hourly, daily, weekly, or monthly, click Schedule Run.

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

With Extract Spark runtime lineage turned on, each run reads runtime lineage events from completed UTC hours only. The first run reads the previous 30 days, and each later run continues from where the previous run stopped. Events from the current hour are read by the next run, so schedule the crawler to match how fresh you need notebook lineage to be. For details, see Spark runtime lineage.

See also​