## Integrate Google Cloud Composer/OpenLineage

URL: https://docs.atlan.com/apps/connectors/lineage/google-cloud-composer-openlineage/how-tos/integrate-google-cloud-composer-openlineage

> To integrate Google Cloud Composer/OpenLineage with Atlan, complete the following steps. To learn more about OpenLineage, refer to [OpenLineage configuration and facets](/product/connections/references/openlineage-configuration-and-facets).

To integrate Google Cloud Composer/OpenLineage with Atlan, complete the following steps. To learn more about OpenLineage, refer to [OpenLineage configuration and facets](https://docs.atlan.com/llms/catalog/connector-capabilities/openlineage-configuration-and-facets/llms.txt).

:::info **Did you know?** 
 For Apache Airflow operators supported for OpenLineage extraction, you can refer to [Airflow's Supported operators](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/supported_classes.html) documentation. To learn how to extract lineage though OpenLineage methods, custom extractors, or manually annotated lineage, see [How to implement OpenLineage in Airflow operators](https://docs.atlan.com/llms/connectors/apache-airflow-openlineage/implement-openlineage-in-airflow-operators/llms.txt). Also, check the [recommended provider package versions for OpenLineage](https://docs.atlan.com/llms/connectors/apache-airflow-openlineage/recommended-provider-package-versions/llms.txt).
:::

## Create an API token in Atlan

Before running the workflow, create an [API token](https://docs.atlan.com/llms/platform/get-started/api-authentication/llms.txt) or configure an [OAuth client](https://docs.atlan.com/llms/platform/get-started/oauth-clients/llms.txt) in Atlan.

## Configure the integration in Atlan

### Select the source in Atlan

To select Google Cloud Composer/OpenLineage as your source, from within Atlan:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Google Cloud Composer Airflow Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Create connection

:::warning

A single connection (namespace) must be used for only one Airflow instance. Using the same connection across multiple instances may cause environment variables to update incorrectly, leading to unexpected behavior.

:::

You will only need to create a connection once to enable Atlan to receive incoming OpenLineage events. Once you have set up the connection, you neither have to rerun the workflow nor schedule it. Atlan will process the OpenLineage events as and when your DAGs run to catalog your Apache Airflow assets.

To configure the Google Cloud Composer/OpenLineage connection, from within Atlan:

1. For _Connection Name_, provide a connection name that represents your source environment. For example, you might use values like `production`,`development`,`gold`, or `analytics`.

2. (Optional) To change the users who are able to manage this connection, change the users or groups listed under _Connection Admins_.

 :::warning

 If you do not specify any user or group, no one will be able to manage the connection - not even admins.

 :::

3. (Optional) For _Host_, enter the URL of your Google Cloud Composer Airflow UI. This will allow Atlan to help you view your assets directly in Google Cloud Composer from the asset profile.

4. (Optional) For _Port_, enter the port number for your Google Cloud Composer Airflow UI.

5. For _Enable OpenLineage Events_, click **Yes** to enable the processing of OpenLineage events or click **No** to disable it. If disabled, new events will not be processed in Atlan.

6. To create a connection, at the bottom of the screen, click the **Create connection** button.

## Configure the integration in Google Cloud Composer

:::info **Did you know?** 
 You will need the Atlan API token and connection name to configure the integration in Google Cloud Composer. This will allow Google Cloud Composer to connect with the OpenLineage API and send events to Atlan.

:::

:::warning

Atlan does not support integrating with Apache Airflow versions older than 2.5.0.

:::

To configure Google Cloud Composer to send OpenLineage events to Atlan:

1. You will need to configure Google Cloud Composer for the integration:
 1. Open your Google Cloud console and navigate to the _[Environments](https://console.cloud.google.com/composer/environments)_ page.
 2. From the list of environments, click the name of your environment. Configure the following: 
 - For Apache Airflow versions 2.7.0 onward, [set override Airflow configuration options](https://cloud.google.com/composer/docs/overriding-airflow-configurations#console):
 1. In the _Environment details_ page, click the **Airflow configuration overrides** tab and then click **Edit**.
 2. In the _Airflow configuration overrides_ form, click the **Add Airflow configuration override** button to specify the first set of values:
 1. For _Section 1_, enter `openlineage`.
 2. For _Key 1_, enter `namespace`.
 3. For _Value 1_, enter the connection name as exactly configured in Atlan.
 3. Click the **Add Airflow configuration override** button to specify the second set of values:
 1. For _Section 2_, enter `openlineage`.
 2. For _Key 2_, enter `transport`.
 3. For _Value 2_, enter the following:

 ```json
 {
 "type": "http", 
 "url": "https://<instance>.atlan.com/events/openlineage/airflow-cloud-composer/", 
 "auth": { "type": "api_key", 
 "api_key": "" 
 } 
 }
 ```

 - For ``, set the API token generated in Atlan.
 - For Apache Airflow versions 2.5.0 onward and prior to 2.7.0, [set environment variables](https://cloud.google.com/composer/docs/how-tos/managing/environment-variables):
 1. In the _Environment details_ page, click the **Environment variables** tab and then click **Edit**.
 2. Add the following environment variable names and corresponding values:
 - `OPENLINEAGE_URL`: points to the service that will consume OpenLineage events - for example, `https://<instance>.atlan.com/events/openlineage/airflow-cloud-composer/`.
 - `OPENLINEAGE_API_KEY`: set the API token generated in Atlan.
 - `OPENLINEAGE_NAMESPACE`: set the connection name as exactly configured in Atlan.
 3. Click **Save** to save your changes.
2. You will also need to install the OpenLineage PyPI package in Google Cloud Composer. To install the OpenLineage PyPI package in your environment:
 1. In the _Environment details_ page, click the **PyPI packages** tab and then click **Edit**.
 2. Click **Add package** to add a custom package.
 3. Under _PyPI packages_, for _Package name_, specify the package name.
 - For Apache Airflow versions 2.7.0 onward:

 ```
 apache-airflow-providers-openlineage
 ```

 - For Apache Airflow versions 2.5.0 onward:

 ```
 openlineage-airflow
 ```

 4. Click **Save** to save your configuration.

## Verify the Atlan connection in Google Cloud Composer

To verify connectivity to Google Cloud Composer:

1. For _Verify connection with Cloud Composer_, click the clipboard icon to copy and run the preflight check DAG on your Google Cloud Composer instance to test connectivity with Atlan. If you encounter any errors after running the DAG, refer to the [preflight checks documentation](https://docs.atlan.com/llms/connectors/apache-airflow-openlineage/preflight-checks-for-apache-airflow/llms.txt).

2. Click **Done** to complete setup.

Once your DAGs have completed running in Apache Airflow, you will see Apache Airflow DAGs and tasks along with lineage from OpenLineage events in Atlan! 🎉

You can also [view event logs](https://docs.atlan.com/llms/governance/access-control/view-event-logs/llms.txt) in Atlan to track and debug events received from OpenLineage.

---