What does Atlan crawl from Google Cloud Composer/OpenLineage?
Once you have integrated Google Cloud Composer/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:
- Status filter - last run status for an asset
- Duration filter - last run duration for an asset
Atlan maps the following assets and properties from Google Cloud Composer/OpenLineage. Asset lineage support depends on the list of operators supported by OpenLineage.
DAGs
Atlan maps DAGs (directed acyclic graphs) from Google Cloud Composer/OpenLineage to its AirflowDAG
asset type.
Source property | Atlan property |
---|---|
schedule_interval | airflowDagSchedule |
timetable | airflowDagScheduleDelta |
tags | airflowTags |
version | airflowRunVersion |
openlineageAdapterVersion | airflowRunOpenLineageVersion |
runid | airflowRunName |
run_type | airflowRunType |
eventTime | airflowRunStartTime |
eventTime | airflowRunEndTime |
eventType | airflowRunOpenLineageState |
query | airflowTaskSql |
group_id | airflowTaskGroupName |
Tasks
Atlan maps tasks from Google Cloud Composer/OpenLineage to its AirflowTask
asset type.
Source property | Atlan property |
---|---|
retries | airflowTaskRetryNumber |
pool | airflowTaskPool |
pool_slots | airflowTaskPoolSlots |
queue | airflowTaskQueue |
priority_weight | airflowTaskPriorityWeight |
trigger_rule | airflowTaskTriggerRule |
operator_class | airflowTaskOperatorClass |
dag_id | airflowDagName |
conn_id | `airflowTaskConnectionId` |
sql | airflowTaskSql |
tags | airflowTags |
version | airflowRunVersion |
openlineageAdapterVersion | airflowRunOpenLineageVersion |
runid | airflowRunName |
run_type | airflowRunType |
eventTime | airflowRunStartTime |
eventTime | airflowRunEndTime |
eventType | airflowRunOpenLineageState |