## Set up Google Dataflow

URL: https://docs.atlan.com/apps/connectors/lineage/google-dataflow/how-tos/set-up-google-dataflow

> Enable lineage on Dataflow jobs and grant Atlan read access to Google Cloud's Data Lineage API.

# Set up Google Dataflow 

The Google Dataflow connector reads job-run lineage that Dataflow publishes to Google Cloud's **Data Lineage API** and creates the jobs, datasets, and lineage in Atlan. Setup has three parts: enable the API, turn on lineage for your Dataflow jobs, and create a read-only service account for Atlan.

## Prerequisites

- A Google Cloud project running Dataflow jobs, and permission to manage IAM and enable APIs on it (for example `roles/owner` or Project IAM Admin).
- Dataflow jobs on **Apache Beam SDK 2.63.0 or later** (lineage isn't reported by older SDKs).

## Enable Data Lineage API

For every project you want Atlan to read lineage from:

### Google Cloud console

APIs & Services → Enable APIs and services → search for **Data Lineage API** → Enable.

### gcloud CLI

```bash
gcloud services enable datalineage.googleapis.com --project=<project_id>
```

## Enable lineage on your Dataflow jobs

Lineage is opt-in **per job** via the `enable_lineage` Dataflow service option:

```bash

# Java / Python pipeline launch

--dataflowServiceOptions=enable_lineage=true # Java
--dataflow_service_options=enable_lineage=true # Python

# Classic or Flex templates

gcloud dataflow jobs run <job> ... \
 --additional-experiments=enable_lineage=true
```

Lineage is captured for these sources and sinks: **Pub/Sub, Cloud Storage, Bigtable, Spanner, Apache Kafka, JDBC**.

:::warning Unsupported sinks

Sources and sinks outside that list—for example **Apache Iceberg** writes—don't produce lineage records. Those jobs still appear in Atlan (with run status), but without dataset edges.

:::

## Create service account for Atlan

The connector is read-only; one project-level role is sufficient:

| Role | Why |
|---|---|
| `roles/datalineage.viewer` | Read processes, runs, and lineage events from the Data Lineage API |

### Google Cloud console

1. IAM & Admin → Service Accounts → **Create service account**.
2. Name it (for example `atlan-dataflow-lineage`) → Create and continue.
3. Grant the role **Data Lineage Viewer** → Done.
4. Open the account → Keys → Add key → **Create new key** → JSON. Keep the downloaded file safe—Atlan needs its contents.

### gcloud CLI

```bash
gcloud iam service-accounts create atlan-dataflow-lineage --project=<project_id>

gcloud projects add-iam-policy-binding <project_id> \
 --member="serviceAccount:atlan-dataflow-lineage@<project_id>.iam.gserviceaccount.com" \
 --role="roles/datalineage.viewer"

gcloud iam service-accounts keys create key.json \
 --iam-account=atlan-dataflow-lineage@<project_id>.iam.gserviceaccount.com
```

To read lineage from multiple projects, grant the same role on each project.

## Configure connector in Atlan

The workflow setup asks for three things:

1. For **Credential**, paste the **Service Account JSON**: the contents of the key file downloaded when creating the service account.
2. For **Connection**, enter the **Connection Name**: the Atlan connection your lineage is published into.
3. For **Configuration**, provide:

| Field | Value |
 |---|---|
 | GCP Project IDs | The projects to poll |
 | Locations | The regions your jobs run in (for example `us-central1`) |

## Verify

Run the workflow once, then check the run logs: you'll see `preflight ok`, per-project scan counts, and `posted OL event` lines. In Atlan, your Dataflow jobs appear as assets named `Dataflow <job-name>` inside the configured connection, with lineage edges to their datasets.

---