Skip to main content
Community Hub

Integrate Generic OpenLineage

TL;DR

Atlan extracts job-level operational metadata from applications sending OpenLineage events and generates job lineage through OpenLineage.

Your AI can read this via Docs MCPInstall MCP →Connect
Public Preview

Atlan supports ingestion of OpenLineage (OL) events from any system that can emit OpenLineage-compliant events. This generic integration lets you connect any OpenLineage source (not limited to Apache Airflow or Spark) to Atlan. To learn more about OpenLineage, refer to OpenLineage configuration and facets.

warning

Atlan currently supports only the HTTP transport mechanism for receiving OpenLineage events.

Once configured, Atlan automatically processes incoming lineage events and catalogs your data workflows and assets.

Prerequisites​

Before setting up the integration, make sure you have:

  • Credentials to authenticate with: either an Atlan API token or an Atlan OAuth client
  • A connection name created in Atlan (Generic OpenLineage)
  • A source system capable of emitting OpenLineage events over HTTP

Choose authentication method​

Atlan accepts two kinds of credentials for OpenLineage events:

MethodCredential you holdToken lifetimeBest for
API tokenA token you paste into your sourceUp to one yearSources that only accept a static bearer token
OAuth clientA client ID and client secret10 minutes, refreshed by your sourceSources that can call a token endpoint

Both are supported. Choose OAuth when your source can request a token, because the credential is short-lived, and you can rotate the secret without editing your source configuration.

Create API token in Atlan​

To authenticate your OpenLineage source with Atlan, you need an API token.

  1. Go to the Atlan Admin Panel
  2. Navigate to API Tokens
  3. Generate a new token

This token serves as the authentication key when sending OpenLineage events.

Send it on every request:

Authorization: Bearer <API_TOKEN>

Create OAuth client in Atlan​

Use an OAuth client to authenticate with short-lived tokens instead of a static one.

  1. Go to the Atlan Admin Panel
  2. Navigate to API Tokens, then OAuth clients
  3. Click New OAuth client
  4. In Role, choose the base role for the client
  5. In Personas, select the personas the client needs
  6. Copy the client ID and client secret

The client secret is shown only once. Store it in a secret manager rather than in source control.

note

Role and personas govern which other Atlan APIs the client can call. Event ingestion is authorized on the client itself, so any OAuth client on your tenant can send OpenLineage events.

Request access token​

Exchange the client ID and secret for an access token using the OAuth 2.0 client credentials grant:

curl -s -X POST "https://<instance>.atlan.com/auth/realms/default/protocol/openid-connect/token" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "grant_type=client_credentials&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>"

The response contains the token and its lifetime in seconds:

{
  "access_token": "<ACCESS_TOKEN>",
  "expires_in": 600,
  "token_type": "Bearer"
}

Send the token on every event request, the same way as an API token:

Authorization: Bearer <ACCESS_TOKEN>

Refresh token​

Access tokens expire after 10 minutes. Your source must request a new token before the current one expires. Request one token, cache it, and refresh shortly before expires_in elapses. Don't request a token for every event.

There is no refresh token. Each request for a new token uses the client ID and secret again.

warning

Never log the response from the token endpoint. It contains the access token.

Send events with access token​

import time, requests

TOKEN_URL = "https://<instance>.atlan.com/auth/realms/default/protocol/openid-connect/token"
EVENTS_URL = "https://<instance>.atlan.com/events/openlineage/generic-openlineage/api/v1/lineage"

_token, _expiry = None, 0.0

def bearer() -> str:
    global _token, _expiry
    if _token and time.time() < _expiry - 120:   # refresh two minutes early
        return _token
    response = requests.post(TOKEN_URL, timeout=10, data={
        "grant_type": "client_credentials",
        "client_id": CLIENT_ID,
        "client_secret": CLIENT_SECRET,
    })
    response.raise_for_status()
    body = response.json()
    _token = body["access_token"]
    _expiry = time.time() + int(body.get("expires_in", 600))
    return _token

requests.post(EVENTS_URL, json=event,
              headers={"Authorization": f"Bearer {bearer()}"})

Rotate and revoke credentials​

  • Rotate an OAuth secret: generate a second secret, roll your source over to it, then delete the old one. Two secrets can be active at once, so there's no downtime.
  • Revoke an OAuth client: disable or delete it in the Admin Panel. Tokens already issued stay valid until they expire, so wait up to 10 minutes.
  • Rotate an API token: generate a new token, update your source, then revoke the old one.

Configure integration in Atlan​

  1. In the top-right corner, click New
  2. Select New workflow
  3. Search for Generic OpenLineage Assets
  4. Click Setup Workflow

Create connection​

A connection represents the namespace under which lineage events are grouped.

  • Connection Name: A name representing the source environment (for example, production, development, analytics)
  • Connection Admins (optional): Assign users or groups who can manage this connection. If no admin is assigned, no one—including Atlan admins—can manage the connection.

Click Create connection to finish.

A single connection can receive OpenLineage events from multiple source systems. However, we recommend creating one connection per source instance—this keeps assets segregated by source in the Atlan UI and makes it easier to debug issues when they arise. Don't create connections with a duplicate name in Generic OpenLineage Assets.

note

You no longer pick a source system (Airflow, Spark, Flink, and so on) when creating the connection. Atlan identifies the source automatically from each incoming event's job.facets.jobType.integration attribute. Events are routed to this connection by matching job.namespace to the connection name—see Event conventions.

Configure your OpenLineage source​

Your OpenLineage producer must send events to Atlan using HTTP transport.

Endpoint​

All OpenLineage events must be sent to:

https://<instance>.atlan.com/events/openlineage/generic-openlineage/api/v1/lineage

Replace <instance> with your Atlan tenant name.

Validate your setup with sample events​

Use the public examples repo to quickly verify your connector is working end-to-end.

Send events with OpenLineage Studio​

OpenLineage Studio is a local app included in the examples repo. Pick an example, review the prefilled OpenLineage payload, and send it to your connection—no .env file and no npm install required. Studio requires Node.js 20 or later.

git clone https://github.com/atlanhq/generic-openlineage-examples.git
cd generic-openlineage-examples/studio
node serve.mjs

Studio opens at http://localhost:5174 and guides you through a one-time setup:

  1. Atlan instance: the part of your tenant URL before .atlan.com
  2. API token: the token generated in Create API token in Atlan. OpenLineage Studio accepts a static token only. To try an OAuth client, request an access token as shown in Request access token, then paste it here, noting that it expires after 10 minutes
  3. Connection: the connection created in Create connection

Select any example, click Send, then click View assets in Atlan to confirm the assets landed.

note

If your network uses a corporate TLS proxy, run node --use-system-ca serve.mjs so that Node.js trusts your organization's certificate authority.

Send events from script​

To send the same events from a terminal or a CI pipeline, use the send_events.py script in the repo:

pip install -r requirements.txt
cp .env.example .env
python send_events.py examples/01_simple_dag

Before running the script, edit .env and fill in OL_ENDPOINT with your event endpoint, API_KEY with the API token generated in Create API token in Atlan or an access token from Request access token, and NAMESPACE with the connection name created in Create connection. For more details, refer to Sending events with the Python CLI.

Verify your event format​

Events are raw OpenLineage RunEvent JSON—no additional envelope needed. Each request body contains a single event.

Minimal required fields:

{
  "eventTime": "2025-01-15T10:00:00.000Z",
  "eventType": "START",
  "producer": "https://my-system.example.com",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
  "run": { "runId": "<uuid>" },
  "job": { "name": "<job-name>", "namespace": "<namespace>" },
  "inputs": [],
  "outputs": []
}

See also​