What does Atlan crawl from Google Dataflow?
Once you have set up Google Dataflow, Atlan reads job-run lineage from Google Cloud's Data Lineage API and maps the following assets and lineage.
Dataflow jobs
Atlan maps each Dataflow job to a process asset named Dataflow <job-name> inside the configured connection.
| Source property | Atlan property |
|---|---|
| Job name | Asset name (Dataflow <job-name>) |
| GCP project ID | Qualified name |
| Location (region) | Qualified name |
| Last run status | Last run status |
Datasets and lineage
For each lineage-supported source and sink, Atlan creates the referenced dataset and draws lineage edges between the datasets and the Dataflow job. Lineage is captured for the following sources and sinks:
- Pub/Sub
- Cloud Storage
- Bigtable
- Spanner
- Apache Kafka
- JDBC
note
Sources and sinks outside the list—for example Apache Iceberg writes—don't produce lineage records. Those jobs still appear in Atlan with run status, but without dataset edges. For details, see Set up Google Dataflow.