## Integrate Apache Airflow/OpenLineage

URL: https://docs.atlan.com/apps/connectors/lineage/apache-airflow-openlineage/how-tos/integrate-apache-airflow-openlineage

> To integrate Apache Airflow/OpenLineage with Atlan, complete the following steps. To learn more about OpenLineage, refer to [OpenLineage configuration and facets](/product/connections/references/openlineage-configuration-and-facets).

To integrate Apache Airflow/OpenLineage with Atlan, complete the following steps. To learn more about OpenLineage, refer to [OpenLineage configuration and facets](https://docs.atlan.com/llms/catalog/connector-capabilities/openlineage-configuration-and-facets/llms.txt).

Atlan also supports other Apache Airflow distributions to enhance your data management and workflow capabilities:

- [Amazon MWAA](https://docs.atlan.com/llms/connectors/amazon-mwaa-openlineage/integrate-amazon-mwaa-openlineage/llms.txt)
- [Astronomer](https://docs.atlan.com/llms/connectors/astronomer-openlineage/integrate-astronomer-openlineage/llms.txt)
- [Google Cloud Composer](https://docs.atlan.com/llms/connectors/google-cloud-composer-openlineage/integrate-google-cloud-composer-openlineage/llms.txt)

:::info **Did you know?** 
 You need the Atlan API token and connection name to configure the integration in Apache Airflow/OpenLineage. This lets Apache Airflow connect with the OpenLineage API and send events to Atlan.

:::

## Create API token in Atlan

Before running the workflow, create an [API token](https://docs.atlan.com/llms/platform/get-started/api-authentication/llms.txt) or configure an [OAuth client](https://docs.atlan.com/llms/platform/get-started/oauth-clients/llms.txt) in Atlan.

## Configure integration in Atlan

### Select source

To select Apache Airflow/OpenLineage as your source, from within Atlan:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Airflow Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Create connection

:::warning

A single connection (namespace) must be used for only one Airflow instance. Using the same connection across multiple instances may cause environment variables to update incorrectly, leading to unexpected behavior.

:::

You only need to create a connection once to enable Atlan to receive incoming OpenLineage events. Once you have set up the connection, you neither have to rerun the workflow nor schedule it. Atlan processes the OpenLineage events as and when your DAGs run to catalog your Apache Airflow assets.

To configure the Apache Airflow/OpenLineage connection, from within Atlan:

1. For _Connection Name_, provide a connection name that represents your source environment. For example, you might use values like `production`,`development`,`gold`, or `analytics`.

2. (Optional) To change the users who are able to manage this connection, change the users or groups listed under _Connection Admins_.

 :::warning

 If you don't specify any user or group, no one can manage the connection - not even admins.

 :::

3. (Optional) For _Host_ and _Port_, enter the URL and port number of your Apache Airflow UI, respectively. This lets Atlan help you view your assets directly in Apache Airflow from the asset profile.

4. For _Enable OpenLineage Events_, click **Yes** to enable the processing of OpenLineage events or click **No** to disable it. If disabled, new events won't be processed in Atlan.

5. To create a connection, at the bottom of the screen, click the **Create connection** button.

## Configure integration in Apache Airflow/OpenLineage

:::info **Did you know?** 
 For Apache Airflow operators supported for OpenLineage extraction, you can refer to [Airflow's Supported operators](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/supported_classes.html) documentation. To learn how to extract lineage though OpenLineage methods, custom extractors, or manually annotated lineage, see [How to implement OpenLineage in Airflow operators](https://docs.atlan.com/llms/connectors/apache-airflow-openlineage/implement-openlineage-in-airflow-operators/llms.txt). Also, check the [recommended provider package versions for OpenLineage](https://docs.atlan.com/llms/connectors/apache-airflow-openlineage/recommended-provider-package-versions/llms.txt).

:::

:::warning

Atlan doesn't support integrating with Apache Airflow versions older than 2.5.0.

:::

### Install OpenLineage package

The OpenLineage package you need depends on your Apache Airflow version. Install the correct package version for compatibility:

1. Identify your Apache Airflow version and the corresponding package from the following table:

 | Airflow version | Required package | Maximum compatible version |
 |-----------------|------------------|---------------------------|
 | 2.5.0 to 2.6.x | `openlineage-airflow` | Latest available |
 | 2.7.0 to 2.7.x | `apache-airflow-providers-openlineage` | 1.10.0 |
 | 2.8.0 to 2.8.x | `apache-airflow-providers-openlineage` | 1.14.0 |
 | 2.9.0 to 2.9.x | `apache-airflow-providers-openlineage` | 2.2.0 |
 | 2.10.0 and later | `apache-airflow-providers-openlineage` | Latest (2.7.3 currently) |
2. Download and install the appropriate package:

 **For Airflow 2.7.0 and later:**
 - Download the compatible version of [`apache-airflow-providers-openlineage`](https://pypi.org/project/apache-airflow-providers-openlineage/) based on the maximum version in the table.
 - Add the package to your `requirements.txt` file:
 ```
 apache-airflow-providers-openlineage==<version>
 ```
 Replace `<version>` with the maximum compatible version for your Airflow version.

 **For Airflow 2.5.0 to 2.6.x:**
 - Download and install the [`openlineage-airflow`](https://pypi.org/project/openlineage-airflow/) library.
 - Add the package to your `requirements.txt` file:
 ```
 openlineage-airflow
 ```

3. Install the package using pip or redeploy your Apache Airflow instance with the updated requirements.

### Configure environment variables

Add the following environment variables to your project's `.env` file:

:::warning

When deploying Apache Airflow on Kubernetes, set these environment variables in both the Scheduler and Triggerer pods for proper integration.

:::

**For Airflow 2.7.0 and later:**

- `AIRFLOW__OPENLINEAGE__NAMESPACE`: set the connection name as exactly configured in Atlan.
- `AIRFLOW__OPENLINEAGE__TRANSPORT`: specify details of where and how to send OpenLineage events in the following JSON string format:

 ```json
 {
 "type": "http", 
 "url": "https://<instance>.atlan.com/events/openlineage/airflow/", 
 "auth": { 
 "type": "api_key", 
 "api_key": ""
 }
 }
 ```

 - Replace `<instance>` with the name of your Atlan instance.
 - Replace `` with the API token generated in Atlan.

**For Airflow 2.5.0 to 2.6.x:**

- `OPENLINEAGE_URL`: points to the service that consumes OpenLineage events - for example, `https://<instance>.atlan.com/events/openlineage/airflow/`.
- `OPENLINEAGE_API_KEY`: set the API token generated in Atlan.
- `OPENLINEAGE_NAMESPACE`: set the connection name as exactly configured in Atlan.

## Verify Atlan connection in Apache Airflow

To verify connectivity to Apache Airflow:

1. For _Verify connection with Airflow_, click the clipboard icon to copy and run the preflight check DAG on your Apache Airflow instance to test connectivity with Atlan. If you encounter any errors after running the DAG, refer to the [preflight checks documentation](https://docs.atlan.com/llms/connectors/apache-airflow-openlineage/preflight-checks-for-apache-airflow/llms.txt).

2. Click **Done** to complete setup.

Once your DAGs have completed running in Apache Airflow, you can see Apache Airflow DAGs and tasks along with lineage from OpenLineage events in Atlan! 🎉

You can also [view event logs](https://docs.atlan.com/llms/governance/access-control/view-event-logs/llms.txt) in Atlan to track and debug events received from OpenLineage.

---