Capture, read, and troubleshoot lineage

||Connect docs via MCP

Lineage in Atlan is generated several different ways, and most "lineage is broken" reports are configuration issues, not bugs. This guide helps you match the right generation mechanism to a source, understand where column-level and cross-system lineage do and don't apply, run an impact analysis before a change, and report lineage health in a way that reflects reality. For the recursive query mechanics behind lineage at scale, follow the metadata-lakehouse (gold-layer) docs—this page is about how to approach lineage, not query internals.

When to use what

Match the mechanism to the source. Don't default to "a connector is needed," and don't default to "it's all custom"—both extremes erode trust.

Situation	Mechanism	Notes
Intra-system table→view, view→view	Crawler (native)	Created during ingestion
Intra-system table→table	Miner (online)	Query-history based; the usual fix for "no table-to-table lineage"
Warehouse → BI (supported pairs)	BI crawler	Cross-system support varies by BI tool; check the supported pairs
Transformation tools (for example, dbt, Fivetran, Airflow)	Transformation connector	Parses job metadata, within and across systems
On-prem / air-gapped / no online pipeline	Offline miner (export query history to object storage)	Intra-system only
No dedicated offline miner exists	Generic miner (query-log based)	Intra-system only; automates vs a manual CSV
Source/dest names same or similar	Name-match generator (regex)	for example, a like-for-like migration
Names differ completely / no connector	Mapping-CSV builder	One process row per source→dest; can autoconsume the CSV from object storage
Full control / code-comfortable	API / SDK	Most flexible; the path for custom asset types
Spark / Airflow / Flink / event-driven	OpenLineage	Real-time; column lineage for SQL/relational sources

Cross-system limits to set expectations on: native cross-system lineage covers only the pairs documented in the lineage docs—check the current list before promising coverage (historically warehouse→BI pairs). Hops outside that list, for example, source→warehouse (Postgres→Snowflake), aren't automatic—use a mapping-CSV builder, a name-match generator, OpenLineage, or the SDK. For multi-hop chains no single connector covers the whole chain: separate what already works from the specific missing hop, and solve the gap, not the whole chain.

Before you build a custom connector, check whether downstream metadata already captures provenance—a custom-metadata field, tag, or playbook can answer "where did this come from?" today, in minutes, versus waiting weeks for a connector. A custom script is a legitimate interim unblock for a deadline, but pair it with a migration plan to a native or builder path—never leave it as the permanent solution.

A miner typically separates two capture streams: a distinction that matters when a security or architecture review objects to SQL ingestion. Data-movement statements (DDL/insert history) drive transformation lineage; usage/query-history (read patterns) drives usage analytics and enrichment context. Confirm the specific capture toggles in the connector/miner reference before committing to them in a review. Where they can be separated, disabling usage capture while keeping transformation lineage on is often enough to clear the objection. When a security or architecture review objects to SQL ingestion, decompose the objection before conceding lineage: if the concern is ingesting SQL at all, disable query-history capture and keep transformation lineage (an offline/export-based miner is the alternative for the most sensitive environments); if the concern is humans reading the ingested SQL, restrict who can see the SQL display field through persona-based access instead. Don't re-escalate a concern the review already cleared—it only stalls momentum.

Column-level lineage

Column-level lineage is supported for relational/SQL sources and for object-store paths mapped through a structured catalog. Native object stores, message queues, and unstructured GCS paths are table-level only.

A column link can be legitimately absent because of transformation-type filtering, not because of a bug. Column lineage is created only for certain transformation types—broadly, direct transformations (identity, calculated, aggregation) and conditional transformations; set-level operations such as JOIN, FILTER, and GROUP BY are excluded as too broad. Confirm the current supported transformation types in the lineage product reference before assuming a defect. So when someone reports "table-level works but the column link is missing," verify the transformation type is direct or conditional, the input source is SQL-based, the input columns exist, and column-name case matches—before assuming a defect.

Object-store lineage (S3 / GCS / ADLS)

Don't crawl individual object-storage files, and don't expect column-level lineage for native object storage: these systems lack schema, so this is a deliberate, documented limitation, not a gap to escalate. Route through a structured layer instead, in this order of preference:

1. Existing metadata catalog (for example, Glue, Unity) → crawl the catalog layer. Quick win—the object-store locations already live in the table metadata.
2. External / Iceberg / Delta tables in the warehouse → crawl them with the native warehouse connector; the external-location property captures the path. Supports column-level lineage.
3. Spark / Airflow / Glue ETL → OpenLineage integration; real-time, column lineage from SQL parsing (objects appear as symlink datasets, not full assets).
4. Prefix-level crawling (last resort) → if you must represent storage, crawl prefixes/folders (not individual files) via a custom package, then build lineage to/from prefixes. Still coarse-grained, still no column-level lineage.

Discovery first: before proposing anything, establish the business need, which personas benefit, object volume and formats, existing catalogs, and the granularity actually required.

Operating practices

Run an impact analysis before any schema change, deprecation, type change, rename, or migration: you need to know what breaks first:

1. Identify the asset (its identifier / qualified name).
2. Query downstream, starting with Level 1 (direct dependencies): that's the highest risk.
3. Time-filter to active assets (updated in the last 7–30 days) to cut noise.
4. Prioritize certified and mission-critical assets.
5. Extract the owner list so you can notify them.
6. Document the blast radius for change control, and plan a rollback.

Sequence lineage before AI enrichment. Enrichment and context-generation quality depends on lineage context—validate lineage first, then enrich. (See Metadata enrichment for why physical-asset enrichment must precede metric and context generation.)

Preview before you generate for any custom-lineage builder or generator—a preview validates the mappings; skipping it risks publishing wrong lineage broadly. Use one regex config rather than one workflow per prefix.

Report valid coverage, not raw coverage

Don't lead with raw coverage

Raw coverage (assets with lineage / all assets) badly understates reality. A tenant with 14% raw coverage can have 97% valid coverage: the same data, a completely different story for stakeholders.

Metric	What it measures	What to report
Raw coverage	Assets with any lineage / all assets	Don't lead with this
Valid coverage	Coverage after excluding assets that legitimately have no lineage—static/origin assets, assets whose parents aren't cataloged yet, unsupported asset types	Lead with this
Completeness	How fully a covered asset's lineage resolves—for example, 16 of 20 query references traced	Weight toward critical assets

Related reads worth running periodically: orphans (no lineage at all) are cleanup candidates, but keep recently-created assets and investigate no-owner ones; circular dependencies may be intentional recursive CTEs; and lineage hubs (the most-connected assets) have the widest blast radius, so prioritize them for monitoring and change management. For the query mechanics behind these reports, follow the gold-layer docs.

Common pitfalls

• Reporting raw coverage as the headline metric instead of valid coverage.
• Treating a lineage gap as a platform bug before ruling out misconfiguration—wrong scope, uncataloged sources, or a miner that never ran.
• Assuming a completed miner/crawler run means lineage is correct: a run can finish clean and produce nothing if its inputs are empty or wrong, or if source assets aren't cataloged. Verify the source assets exist in Atlan first.
• Crawling object-store files, or promising object-store column lineage.
• Running a custom-lineage generate at scale without a preview pass first.
• Conflating lineage existence with measures/metadata being cataloged: they're independent connector behaviors; verify, don't assume.
• Answering with documentation alone when a team lacks the access or confidence to execute—run a hands-on working session with them instead.
• Speculating on edge cases (linked-server parsing, cross-engine query transfer) instead of testing empirically. If you gave a "that it works" answer and it doesn't, say so and verify—don't double down.
• Leaving a gap acknowledged without a named owner and a committed next-step date.

Troubleshooting

Work the tree in order before escalating—the recurring failure mode is declaring a bug before ruling out configuration.

1. Crawler vs miner. Is the missing lineage table→table? Crawlers only create table→view / view→view. Table→table needs a miner. Did the right miner run?
2. Did the miner actually produce lineage? An error-free run doesn't mean lineage exists—the miner can run clean and produce nothing if its inputs are empty or wrong.
3. Are the source/upstream assets cataloged in Atlan? Lineage can't point to assets that aren't there. Missing parents are "validly no coverage," not a bug.
4. Is the scope correct? Wrong workspace/project/dataset scope, or a connector pulling from unexpected projects. Confirm how a connector's metadata filters actually scope before relying on them for isolation.
5. Credentials / rotation. Rotated or insufficient credentials silently drop query history.
6. Query-history gaps. Job failures leave holes; some read-only replicas can't enable a query store (use an alternative activity source); if the same transformation runs in both dev and prod, later runs can overwrite the lineage of a shared materialized object—re-run the miner over the full historical query window to restore it.
7. Then escalate: open a support ticket scoped with screenshots, the specific affected assets, what you already ruled out, and which lineage you actually want displayed.

Known limitation classes—disclose these upfront rather than escalating as bugs: object-store column-level lineage, non-relational column lineage (message queues, native object stores), stored-procedure internals (lineage can stop at a CALL), transformation-type filtering (JOIN/FILTER/GROUPBY column links), and some connector-specific execution patterns that extract metadata but not lineage. Verify a genuine defect with support before promising a fix; confirm capability facts against the connector docs rather than from memory.

Cross-connection stitching: if you split one source type into several connections, the miner stitches lineage across them once both are crawled and mined and both endpoints are present. If lineage still stops at the boundary after both are mined, open a ticket with the specific source→target pair.

Mass archival or reappearing assets: the deletion-threshold circuit breaker fails a workflow when a high percentage of a connection's assets are marked for deletion—confirm the threshold before a large or first-time crawl. On reappearance, soft-deleted assets restore with the same identifier and their business metadata and crawled lineage intact; only purged assets get new identifiers. Monitoring and recovery: there is no built-in alert on deletion percentage, so if archival events matter to you, set up change notifications (a marketplace package can poll the audit API and notify on archive/delete events) or run a periodic archived-assets query after major crawls. For recovery, re-running the workflow against the source is the best path—it restores current state from the system of record. Restoring from a platform metastore backup is a last resort only: it rolls back to the snapshot and loses all enrichment made since.

Slow lineage queries at scale are usually a known query-engine ceiling on deep, high-fan-out graphs—name it as a platform ceiling and route to the gold-layer docs for the resolution options rather than handing over a workaround that makes the performance problem yours to own.

Related

• Connectors and ingestion: crawler/miner sequencing and connection structure that lineage depends on.
• Metadata enrichment: why lineage must precede AI enrichment.
• Metadata-lakehouse (gold-layer) docs—the recursive query model, impact-analysis query mechanics, and coverage-report generation.