Build lineage from CSV

App

You can use the Lineage Builder app to import lineage relationships between assets from a CSV file. This is useful for creating lineage connections that include transformation processes, such as dbt models, ETL workflows, or custom data pipelines that aren't automatically discovered by Atlan's connectors.

Prerequisites

Before you begin, make sure you have:

• Access to the Lineage Builder app. You can verify this by searching for Lineage Builder in the Atlan marketplace. If you don't have access, contact Atlan support or your Atlan customer team to request it.
• A CSV file containing lineage data. The file must follow the format described in the Lineage builder CSV file section of the Lineage Builder reference. You can download a sample CSV file to understand the required structure.
• All connections referenced in your CSV file must already exist in Atlan. The app doesn't create connections automatically.
• Source and target assets that you want to link, or a plan for how to handle assets that don't exist yet.

Setup workflow

This guide walks through importing lineage from a CSV file that includes transformation processes, such home-grown data processing connecting moving data from staging tables to mart tables. This is a common use case for data teams who want to document their transformation pipelines in Atlan.

1. In your Atlan workspace, go to the homepage and click New workflow in the top navigation bar.

2. Search for Lineage Builder, and then select Set up workflow.

3. In the Workflow name field, enter a descriptive name such as:

   staging-to-mart-lineage

4. In the Source section, select Direct upload and click Choose file to upload your CSV file from your local system. Review your CSV file to make sure it includes all required columns. Each row must contain:

   • Source asset details: Source Type, Source Connector, Source Connection, Source Identity, Source Name
   • Target asset details: Target Type, Target Connector, Target Connection, Target Identity, Target Name
   • Transformation process details: Transformation Connector, Transformation Connection, Transformation Identity, Transformation Name

   To upload a CSV from object storage, refer to Source configuration.

5. In the Options section, set Unknown asset handling to Skip them. This ensures that lineage is only created for assets that already exist in Atlan, maintaining data quality and preventing orphaned relationships.

   To learn how to handle missing assets using Create partial assets or Create full assets, see the Unknown asset handling options.

6. Leave the remaining options at their default values:

   • Fail on errors: Set to No to skip invalid rows and continue processing the rest of the file.
   • Case-sensitive match for assets: Set to No for case-insensitive matching.
   • Field separator: Leave at the default value of , (comma), which is the standard CSV separator.
   • Batch size: Leave at the default value of 20.

7. Run the workflow. The app processes each row in your CSV file and creates lineage relationships between the source and target assets, with the transformation process connecting them.

8. Verify the lineage was created correctly by opening one of the source or target assets in Atlan and viewing the lineage graph to confirm the connections appear with transformation processes visible.

You can configure advanced options such as batch processing, error handling, and asset matching behavior. For complete details on all configuration options, see the Lineage Builder reference.

Need help?

If you have any issues related to configuring the app, contact Atlan support.

See also

• Lineage Builder: Complete configuration reference including CSV file format, all options, and supported cloud storage providers.
• What are processes: Learn how transformation processes appear in lineage graphs.
• What are partial assets: Understand when and how to use partial assets for lineage.