Skip to main content

Atlan vs Databricks lineage

This FAQ addresses common questions about how Atlan lineage differs from Databricks native lineage.

How does Atlan lineage differ from Databricks native lineage?​

Atlan filters lineage to show only valid relationships. Databricks native lineage may display lineage across different time periods, which can include historical relationships.

For example, if a table is recreated twice in a single day from two different source tables, Databricks shows both sets of lineage edges. Atlan shows only how data flows into the recreated table, providing a cleaner view of valid data flow relationships.

Why does Atlan lineage look different from what I see in Databricks?​

Atlan filters lineage to show only relationships that are valid. When a table is recreated using CREATE OR REPLACE commands, Atlan removes pre-existing lineage and starts fresh with the recreated table structure. This ensures lineage reflects valid relationships for your data transformations, not historical operations on tables that may have been completely restructured.

Does Atlan use the same lineage data as Databricks?​

Atlan analyzes the same Databricks system tables that Databricks uses, including system.query.history, system.access.table_lineage, and system.access.column_lineage. However, Atlan applies different logic to filter and present lineage, focusing on valid relationships rather than historical completeness.

See also​