What does Atlan crawl from Apache Spark/OpenLineage?
Once you have integrated Apache Spark/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:
- Status filter - last run status for an asset
- Duration filter - last run duration for an asset
Atlan maps the following assets and properties from Apache Spark/OpenLineage. Asset lineage support depends on the data sources that OpenLineage supports.
Jobs
Atlan maps jobs from Apache Spark to its SparkJob asset type. Atlan also supports column-level lineage for Spark jobs.
| Source property | Atlan property |
|---|---|
appName | sparkAppName |
master | sparkMaster |
OpenLineage metadata
Atlan reports OpenLineage operational metadata for Spark jobs.
| Atlan property | Description |
|---|---|
sparkRunVersion | Spark runtime version |
sparkRunOpenLineageVersion | OpenLineage library version |
sparkRunStartTime | job start time |
sparkRunEndTime | job end time |
sparkRunOpenLineageState | status of the job |