Skip to main content
Community Hub

Preflight checks for Apache Airflow

TL;DR

Learn about preflight checks for apache airflow.

Your AI can read this via Docs MCPInstall MCP →Connect

Before running your DAGs in Apache Airflow, Amazon MWAA, Astronomer, or Google Cloud Composer, you can run a preflight check DAG in your Apache Airflow instance to perform the necessary technical validations.

The preflight check DAG:

  • Neither collects nor transmits any sensitive data during the validation process, ensuring the security of your integration.
  • In case of any errors, it will provide detailed feedback for troubleshooting, including error codes and next steps.
  • Includes a retry mechanism for the API call to handle temporary network issues or server unavailability.

Preflight checks​

The preflight check DAG performs the following steps to validate your Atlan and OpenLineage setup:

  1. Collects environment variables - to verify OpenLineage-related environment variables set during the configuration of your Apache Airflow, Amazon MWAA, Astronomer, or Google Cloud Composer instance. These variables can vary depending on your Apache Airflow version.
  2. Validates the OpenLineage library installation - to check whether the openlineage-python library has been installed and identify its version. This ensures that the necessary library for sending OpenLineage events to Atlan is in place.
  3. Sends API call for validation - with the information collected in the previous steps, the preflight check DAG makes a POST request to Atlan's preflight check endpoint. This is to confirm that there are no network issues or configuration errors obstructing the communication.

For example, the payload sent for validation looks like this:

{
    "connector_type": "airflow-mwaa",
    "version": "2.5.0", // Airflow version
    "ol_namespace": "staging-mwaa", // Environment variable
    "ol_endpoint": "https://<host>/events/openlineage/airflow-mwaa/", // Environment variable
    "ol_version": "1.8" // Installed OpenLineage library version
}

Note that Atlan will conduct some additional validations on the Atlan server using the provided data. This is to ensure that the integration is successful. If successful, the DAG will succeed.

Composite transports

If your OpenLineage transport is a composite transport—which sends events to several backends at once, as the Astronomer setup does—the preflight check finds the transport inside it that sends to Atlan and validates that one. A composite transport has no URL or API token of its own.

That means:

  • ol_endpoint in the payload is the URL of the Atlan transport inside the composite, not of the composite itself.
  • If a transform transport wraps the Atlan transport, ol_namespace is its new_job_namespace value, because that's the namespace Atlan actually receives. This value must match the connection name as configured in Atlan.
  • Errors about the transport name the OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__* variables instead of AIRFLOW__OPENLINEAGE__TRANSPORT.

The preflight check identifies the Atlan transport by the name atlan, so keep that name as the setup instructions give it.

Check Apache Airflow DAG logs​

To check Apache Airflow DAG logs:

  1. Open your Apache Airflow homepage.
  2. From the homepage, navigate to the AtlanOpenLineageConnectionTest DAG. Under the Runs column, click the failed run, circled in red.
  3. On the List Dag Run page, under the Run Id column, click the latest or top most failed run ID.
  4. From the corresponding screen, click the run_ol_preflight_check task.
  5. From the tabs along the top of the Task Instance popup, click Log to view logs.
  6. On the Log page, scroll down to the Exception section to view the error code and message. Refer to the troubleshooting guide below to make the necessary changes.

For other distributions, refer to Amazon MWAA, Astronomer, or Google Cloud Composer documentation for more details.

Troubleshoot errors​

Missing environment variables​

Environment variable {var} is missing. Please set it before running the DAG.

Ensure that the required environment variables are set in your environment. These variables can vary depending on your Apache Airflow version:

  • For Apache Airflow versions 2.7.0 onward: AIRFLOW__OPENLINEAGE__NAMESPACE and AIRFLOW__OPENLINEAGE__TRANSPORT.
  • For Apache Airflow versions 2.5.0 onward and prior to 2.7.0: OPENLINEAGE_URL, OPENLINEAGE_API_KEY, and OPENLINEAGE_NAMESPACE.

OpenLineage is disabled​

OpenLineage is disabled in airflow.cfg: openlineage.disabled

OpenLineage is switched off in your Apache Airflow configuration, so the listener never starts and no events are emitted, whatever transport you configure.

  • On Cloud Composer 3, this is the default: Cloud Composer sets [openlineage] disabled to True unless its own Cloud Data Lineage integration is enabled. Add a third Airflow configuration override, section openlineage, key disabled, value False, as described in how to integrate Google Cloud Composer/OpenLineage. Cloud Composer 2 doesn't need this.
  • Anywhere else, remove the [openlineage] disabled override from your Apache Airflow configuration, or set it to False.
OpenLineage is disabled due to the environment variable OPENLINEAGE_DISABLED

Set OPENLINEAGE_DISABLED to false, or remove it.

OpenLineage package isn't installed​

`apache-airflow-providers-openlineage` is not installed- setup docs at ...
`openlineage-airflow` is not installed- setup docs at ...
  • The required OpenLineage package isn't present. Which one you need depends on your Apache Airflow version: apache-airflow-providers-openlineage for 2.7.0 onward, openlineage-airflow for 2.5.0 to 2.6.x. Add it to your requirements.txt and redeploy.

OpenLineage isn't accessible​

OpenLineage provider is not accessible
OpenLineage is not accessible
  • The package is installed but Apache Airflow can't import it. This usually means the installed version isn't compatible with your Apache Airflow version—some provider versions declare support for an Apache Airflow release and then fail to load on it.
  • Install a version known to work with your Apache Airflow version. See recommended provider package versions.

Apache Airflow version is too old​

Atlan does not support Airflow versions <2.5.0. Please upgrade to a supported version.
  • Upgrade Apache Airflow to 2.5.0 or later. This check runs in the DAG itself, before it contacts Atlan.

Transport is missing​

Transport is missing, set variable - AIRFLOW__OPENLINEAGE__TRANSPORT.
Namespace is missing, set variable - AIRFLOW__OPENLINEAGE__NAMESPACE.
  • On Google Cloud Composer, no namespace value was found. Add the Airflow configuration override with section openlineage and key namespace, set to the connection name as exactly configured in Atlan.

OpenLineage is missing configuration​

OpenLineage is missing configuration, please refer to the OL setup docs.
  • No usable transport was found, so no events can be sent. The preflight check reports one of the following alongside it, to narrow down why:
OpenLineage is disabled because required config/env variables are not set.
Transport value found: `{value}`
  • A transport value exists but OpenLineage couldn't build a working transport from it. Compare it against the format your setup instructions give.
Config file is empty or does not exist: `{path}`
  • [openlineage] config_path points at a file that's missing or empty. Clear the setting, or put a valid configuration at that path.
OPENLINEAGE_URL and OPENLINEAGE_ENDPOINT are set to: {url}
  • These legacy variables are set on an Apache Airflow version that expects the newer configuration. For 2.7.0 onward, use AIRFLOW__OPENLINEAGE__TRANSPORT instead.

Error building transport​

There was an error when trying to build transport.
  • OpenLineage raised an error while constructing the transport from your configuration. Check the JSON structure and the variable names against your setup instructions.

Invalid OpenLineage configuration​

Invalid OpenLineage configuration, Error Code: {code}, Error Message: {message}, Setup Docs: ...
  • Atlan received the preflight request and rejected the configuration. The error code identifies the reason—see the sections below for unsupported_airflow_version, connection_not_found, invalid_openlineage_endpoint, unsupported_ol_version and connection_fetching_failed.

API token problems​

API key is missing. Please set the API key using Atlan Docs - ...
  • No API token was found on the transport. Add auth to your transport configuration, with type set to api_key and api_key set to the token generated in Atlan.
API key cannot contain whitespace. Please set the API key using Atlan Docs - ...
  • The token has a space, tab, or newline in it. This is almost always a copy-paste artifact—the token wrapped across lines. Re-copy it as a single line.
Invalid API key format. Please set the API key using Atlan Docs - ...
  • The value isn't shaped like an Atlan API token. Tokens are JSON Web Tokens: three base64url segments separated by dots. Generate a fresh API token and copy the whole value.
Invalid API key. Please update your API key using Atlan Docs - ...
  • Atlan rejected the token with a 401. It's well-formed but not valid—it may have been revoked, expired, or belong to a different Atlan instance than the one in your OpenLineage URL.
Request Header Fields Too Large. API key is too long and incorrect, please update using docs - ...
  • Atlan rejected the request with a 431. The token is far longer than a real one, which usually means extra text was copied along with it.

Invalid transport configuration​

warning

For all errors related to the environment variable configuration for AIRFLOW__OPENLINEAGE__TRANSPORT, Atlan recommends paying attention to the JSON structure and adhering to the expected schema. You will need to ensure that the string values are correctly formatted, and the dictionaries contain the correct types and necessary keys.

The following errors are related to the AIRFLOW__OPENLINEAGE__TRANSPORT configuration for Apache Airflow versions 2.7.0 onward. If you use a composite transport, the same errors name the OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__* variables instead—correct those in the same way.

Transport isn't valid JSON​

"'AIRFLOW__OPENLINEAGE__TRANSPORT' does not contain valid JSON: {reason}."
  • The variable holds text that isn't parseable JSON—most often a missing comma, brace, or quote. The reason names the character position where parsing stopped.

Transport isn't JSON object​

"'AIRFLOW__OPENLINEAGE__TRANSPORT' must be a JSON object, got {type}."
  • The variable must hold a single JSON object, { ... }. A JSON array or a bare string isn't accepted.

Missing keys in 'transport_info'​

"'{key}' is missing, update variable - AIRFLOW__OPENLINEAGE__TRANSPORT."
  • Ensure that the specified key is present in the AIRFLOW__OPENLINEAGE__TRANSPORT JSON structure.

Incorrect type for keys in 'transport_info'​

"'{key}' must be of type {expected_type.name}, update variable - AIRFLOW__OPENLINEAGE__TRANSPORT."
  • Update the type of specified key in the AIRFLOW__OPENLINEAGE__TRANSPORT JSON structure to match the expected type.

White space in impermissible field​

"'{key}' cannot contain whitespace, update variable - AIRFLOW__OPENLINEAGE__TRANSPORT."
  • Remove any white space from the specified key value in the AIRFLOW__OPENLINEAGE__TRANSPORT JSON structure.

Empty or white space in fields​

"'{key}' cannot be empty or just whitespace. update variable - AIRFLOW__OPENLINEAGE__TRANSPORT."
  • Ensure that the specified key in the AIRFLOW__OPENLINEAGE__TRANSPORT JSON structure is neither empty nor does it contain any white space.

Composite transport errors​

These errors apply when your OpenLineage transport is a composite transport, as the Astronomer setup uses.

No transport points to Atlan​

None of the transports inside the composite OpenLineage transport point to Atlan. Add a transport
named `atlan` sending to `https://<host>/events/openlineage/<connector_type>/`. Found: [...]
  • The composite transport was built successfully but none of the transports inside it sends to Atlan. The error lists the transport names it did find.
  • Add the OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__* variables exactly as your setup instructions give them, and keep the name atlan.

No HTTP transport in composite​

The composite OpenLineage transport has no HTTP transport in it.
  • Every transport inside the composite sends somewhere other than over HTTP. Atlan needs an HTTP transport; check OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__TYPE is set to http.

Composite transport couldn't be built​

The composite OpenLineage transport could not be built: {error}.
  • OpenLineage rejected the configuration before Atlan can inspect it. The error names the reason—for example, `url` key not passed to HttpConfig means OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__URL is missing.

Different transport was built​

A composite OpenLineage transport is configured but a `{type}` transport was built instead.
  • Your configuration asks for a composite transport but OpenLineage produced something else, which usually means another setting is taking precedence. If AIRFLOW__OPENLINEAGE__TRANSPORT is set, it overrides the OPENLINEAGE__TRANSPORT__* variables—clear it, or move your whole composite configuration into it.

Namespace is missing​

Namespace is missing. Set the Atlan connection name either as the Atlan transport's
`transformer_properties.new_job_namespace` or as variable - AIRFLOW__OPENLINEAGE__NAMESPACE.
  • With a composite transport, either source is accepted, but one of them has to carry the connection name exactly as configured in Atlan. If a transform transport wraps the Atlan transport, its new_job_namespace is the value Atlan receives and it takes precedence.

Unsupported OpenLineage client version​

The `transform` OpenLineage transport requires `openlineage-python` >=1.33.0. Installed version: `1.25.0`.
  • The composite and transform transports are features of the openlineage-python client, not of the Apache Airflow provider package, so the provider version alone doesn't tell you whether they're available. Minimum client versions: 1.23.0 for a composite transport, 1.33.0 for a transform transport.
  • Some distributions pin an older client than this. Upgrade it with pip install --upgrade 'openlineage-python>=1.33.0', or add the pin to your requirements.txt and redeploy.

Provider version doesn't support transport variables​

Configuring the OpenLineage transport through `OPENLINEAGE__TRANSPORT__*` variables requires
`apache-airflow-providers-openlineage` >=2.6.0 (which needs Airflow >=2.10).
  • Before version 2.6.0 the provider package doesn't read the OPENLINEAGE__TRANSPORT__* variables at all. It treats OpenLineage as unconfigured, never starts the listener, and emits no events—so this setup fails silently on older providers.
  • Either upgrade Apache Airflow and the provider package, or pass the whole transport as JSON in AIRFLOW__OPENLINEAGE__TRANSPORT instead, which every supported provider version reads.

Provider package is out of date​

`apache-airflow-providers-openlineage` is out of date. Installed: `2.2.0`, latest compatible: `2.8.0`.
`apache-airflow-providers-openlineage` is out of date. Installed: `2.9.1`, minimum supported on
Airflow `3.2.2`: `2.9.2`.
  • The first form appears on Apache Airflow versions that have a maximum compatible provider version—install exactly that version. The second appears on newer Apache Airflow versions, which have no maximum: install the named minimum or anything newer.
  • See recommended provider package versions.
  • The provider package's own metadata isn't a reliable guide here: some versions declare support for an Apache Airflow release and then fail to load on it, which is why the preflight check carries its own tested versions.

Configuration is being read from file​

File found at `openlineage.yml` (top-level keys: ['transport'], transport.type: `http`).
OpenLineage is taking its configuration from this file instead of the Atlan variables.
  • An openlineage.yml file exists—in your Apache Airflow working directory, in ~/.openlineage/, or at the path in [openlineage] config_path—and OpenLineage prefers it over the variables you set for Atlan.
  • Remove the file, or clear [openlineage] config_path, and run the preflight check again.
  • The preflight check reports only the structure of that file and never its contents, because it can contain an API token.

Network permission error​

ERROR - Failed to emit OpenLineage eventHTTPSConnectionPool(host='<instance>.atlan.com', port=443):Max retries exceeded with url: /events/openlineage/airflow-cloud-composer/api/v1/lineage
  • This error may result from the firewalls or VPC on which your Apache Airflow instance is hosted. Contact your network team to update the network permissions. This will allow your Airflow instance to make API calls to the URL mentioned in the error message.

Unsupported Apache Airflow version​

{
 "status":"fail",
 "error_code":"unsupported_airflow_version",
 "error_message":"Minimum supported version is 2.5.0, you are using 2.4"
}
  • Atlan does not support integrating with Apache Airflow versions older than 2.5.0. Upgrade your Apache Airflow version to 2.5.0 or above. This will allow OpenLineage to push metadata to Atlan.

Connection not found​

{
 "status":"fail",
 "error_code":"connection_not_found",
 "error_message":"<connection_name> is not present on Atlan."
}
  • The connection name set in your environment variables does not match the connection name created in Atlan. Create a new connection or use an existing connection name.

Invalid OpenLineage endpoint​

{
 "status":"fail",
 "error_code":"invalid_openlineage_endpoint",
 "error_message":"Your OL endpoint should be: https://<instance>.atlan.com/events/openlineage/airflow/ || received: https://<instance>.atlan.com/events/openlineage/airflow/haha please update this."
}
  • Update the environment variable containing the OpenLineage URL to the expected URL in the error message.

Unsupported OpenLineage version​

{
 "status":"fail",
 "error_code":"unsupported_ol_version",
 "error_message":"Minimum supported version is 1.2.0, you are using 1.1"
}
  • Install the latest version of the openlineage-airflow library - 1.8.0 and above. The preflight check DAG will fail if you're using any OpenLineage version older than 1.2.0.

Connection fetching failed​

{
 "status":"fail",
 "error_code":"connection_fetching_failed",
 "error_message":"Atlas Error: Failed to fetch connections - Post \"http://localhost:21000/api/atlas/v2/search/indexsearch\\": dial tcp [::1]:21000: connect: connection refused
}
Did you know?

If you continue to encounter any issues, Atlan recommends enabling debug logging for OpenLineage using the OPENLINEAGE_CLIENT_LOGGING=DEBUG environment variable. Run your DAGs again and then share the debug log with Atlan support for troubleshooting.