Set up Google Dataflow Private Preview

||Connect docs via MCP

The Google Dataflow connector reads job-run lineage that Dataflow publishes to Google Cloud's Data Lineage API and creates the jobs, datasets, and lineage in Atlan. Setup has three parts: enable the API, turn on lineage for your Dataflow jobs, and create a read-only service account for Atlan.

Prerequisites

• A Google Cloud project running Dataflow jobs, and permission to manage IAM and enable APIs on it (for example roles/owner or Project IAM Admin).
• Dataflow jobs on Apache Beam SDK 2.63.0 or later (lineage isn't reported by older SDKs).

Enable Data Lineage API

For every project you want Atlan to read lineage from:

Google Cloud console

APIs & Services → Enable APIs and services → search for Data Lineage API → Enable.

gcloud CLI

gcloud services enable datalineage.googleapis.com --project=<project_id>

Enable lineage on your Dataflow jobs

Lineage is opt-in per job via the enable_lineage Dataflow service option:

# Java / Python pipeline launch
--dataflowServiceOptions=enable_lineage=true      # Java
--dataflow_service_options=enable_lineage=true    # Python

# Classic or Flex templates
gcloud dataflow jobs run <job> ... \
  --additional-experiments=enable_lineage=true

Lineage is captured for these sources and sinks: Pub/Sub, Cloud Storage, Bigtable, Spanner, Apache Kafka, JDBC.

Unsupported sinks

Sources and sinks outside that list—for example Apache Iceberg writes—don't produce lineage records. Those jobs still appear in Atlan (with run status), but without dataset edges.

Create service account for Atlan

The connector is read-only; one project-level role is sufficient:

Role	Why
roles/datalineage.viewer	Read processes, runs, and lineage events from the Data Lineage API

Google Cloud console

1. IAM & Admin → Service Accounts → Create service account.
2. Name it (for example atlan-dataflow-lineage) → Create and continue.
3. Grant the role Data Lineage Viewer → Done.
4. Open the account → Keys → Add key → Create new key → JSON. Keep the downloaded file safe—Atlan needs its contents.

gcloud CLI

gcloud iam service-accounts create atlan-dataflow-lineage --project=<project_id>

gcloud projects add-iam-policy-binding <project_id> \
  --member="serviceAccount:atlan-dataflow-lineage@<project_id>.iam.gserviceaccount.com" \
  --role="roles/datalineage.viewer"

gcloud iam service-accounts keys create key.json \
  --iam-account=atlan-dataflow-lineage@<project_id>.iam.gserviceaccount.com

To read lineage from multiple projects, grant the same role on each project.

Configure connector in Atlan

The workflow setup asks for three things:

1. For Credential, paste the Service Account JSON: the contents of the key file downloaded when creating the service account.
2. For Connection, enter the Connection Name: the Atlan connection your lineage is published into.
3. For Configuration, provide:

Field	Value
GCP Project IDs	The projects to poll
Locations	The regions your jobs run in (for example us-central1)

Verify

Run the workflow once, then check the run logs: you'll see preflight ok, per-project scan counts, and posted OL event lines. In Atlan, your Dataflow jobs appear as assets named Dataflow <job-name> inside the configured connection, with lineage edges to their datasets.