## What does Atlan crawl from Apache Spark/OpenLineage?

URL: https://docs.atlan.com/apps/connectors/lineage/apache-spark-openlineage/references/what-does-atlan-crawl-from-apache-spark-openlineage

> Atlan maps the following assets and properties from Apache Spark/OpenLineage. Asset lineage support depends on the data sources that OpenLineage supports.

Once you have [integrated Apache Spark/OpenLineage](https://docs.atlan.com/llms/connectors/apache-spark-openlineage/integrate-apache-spark-openlineage/llms.txt), you can [use connector-specific filters](https://docs.atlan.com/llms/catalog/discovery/use-the-filters-menu/llms.txt) for quick asset discovery. The following filters are currently supported:

- Status filter - last run status for an asset
- Duration filter - last run duration for an asset

Atlan maps the following assets and properties from Apache Spark/OpenLineage. Asset lineage support depends on the data sources that OpenLineage supports.

## Jobs

Atlan maps jobs from Apache Spark to its `SparkJob` asset type. Atlan also supports column-level lineage for Spark jobs.

| Source property | Atlan property | Description |
|---|---|---|
| `job.name` | `name` | Name of the Spark job |
| - | `qualifiedName` | Unique identifier for the job in Atlan |
| Derived from job.name | `sparkAppName` | Name of the Spark application (substring before first '.') |
| `spark.master` | `sparkMaster` | Spark master URL (for example, yarn, local, and more.) |
| - | `connectionQualifiedName` | Unique identifier for the connector instance |
| - | `connectorName` | Name of the connector instance |

## OpenLineage metadata

Atlan reports OpenLineage operational metadata for Spark jobs.

| Source | Atlan property | Description |
|---|---|---|
| `run.runId` | `sparkRunId` | Unique run identifier |
| `run.facets.spark_version.spark-version` | `sparkRunVersion` | Spark runtime version |
| `run.facets.spark_version.openlineage-spark-version` | `sparkRunOpenLineageVersion` | OpenLineage library version |
| `START` event timestamp | `sparkRunStartTime` | Job start time |
| `COMPLETE`/`ABORT`/`FAIL` event timestamp | `sparkRunEndTime` | Job end time |
| Final event type | `sparkRunOpenLineageState` | Status of the job (COMPLETE, FAIL, ABORT) |

---