Set up authentication for Google BigQuery

✦TL;DR

Set up the service account Atlan connects to Google BigQuery as, and choose between a service account JSON key and Workload Identity Federation.

||

|Connect

Two paths: a service account JSON key (Path A - simplest) or Workload Identity Federation (Path B - keyless). Both start with the same first phase: creating the robot user and its permissions.

Phases at glance

Phase	What happens	Owner	Typical time
1. Prepare (both paths)	Enable APIs, create role + service account	Google Cloud admin	20 minutes
A2. Key file	Generate JSON key, paste into Atlan	Google Cloud admin + Atlan User	15 minutes
B2. WIF trust	OAuth client in Atlan; pool + provider in Google Cloud	Atlan User + Google Cloud admin	~1 hour
3. Verify	Test connection + preflight checks	Atlan User	10 minutes

Phase 1 - Prepare the Google Cloud side

Owner: Google Cloud admin

1a. Enable the Cloud Resource Manager API

Atlan's preflight checks call this API to verify permissions. If it's disabled you'll hit a misleading "Unable to connect to the configured BigQuery instance" error at setup even though everything else is correct - the most common preventable setup failure.

gcloud services enable cloudresourcemanager.googleapis.com \
  --project=example-project

1b. Create a custom role with Atlan's permissions

Grant only what you'll use - see Set up Google BigQuery for the complete permission reference:

Capability	Permissions
Metadata crawling (required)	bigquery.datasets.get, bigquery.datasets.getIamPolicy, bigquery.jobs.create, bigquery.routines.get, bigquery.routines.list, bigquery.tables.get, bigquery.tables.getIamPolicy, bigquery.tables.list, bigquery.readsessions.create, bigquery.readsessions.getData, bigquery.readsessions.update, resourcemanager.projects.get
Data preview & querying (optional)	bigquery.tables.getData, bigquery.jobs.get, bigquery.jobs.listAll, bigquery.jobs.update
Query history mining (optional)	bigquery.jobs.listAll, bigquery.jobs.get
Tags crawling (optional)	resourcemanager.tagKeys.list, resourcemanager.tagValues.list, datacatalog.taxonomies.list, datacatalog.taxonomies.get
External Delta Lake tables (optional)	storage.objects.get, storage.objects.list

Two permission traps

Delta-format external tables: BigQuery itself reads Delta transaction logs from Cloud Storage, so without the two storage.* permissions the crawl fails on exactly those tables with a confusing "Permission denied while globbing file pattern" error.

Crawling projects other than the service account's home project: bind the role in both the home project and every target project. Atlan's test connection verifies permissions against the home project - a role bound only on the target project passes nothing. See Special cases.

1c. Create the service account

gcloud iam service-accounts create atlan-user \
  --description="Atlan Service Account to extract metadata" \
  --display-name="Atlan User"

# Attach your custom role (repeat per target project):
gcloud projects add-iam-policy-binding example-project \
  --member="serviceAccount:atlan-user@example-project.iam.gserviceaccount.com" \
  --role="projects/example-project/roles/atlanUserRole"

Path A, Phase 2 - Service account JSON key

Owner: Google Cloud admin + Atlan User

1. Google Cloud admin - In IAM & Admin → Service Accounts → Keys, create a JSON key and hand the file to whoever sets up the connection, over a secure channel.
2. Atlan User - In the crawler setup choose Service Account auth and fill in: Project ID (the project_id field of the JSON), Service Account JSON (paste the whole file), Service Account Email (the client_email field).

Key rotation breaks schedules silently

If anyone rotates or deletes this key in Google Cloud, every scheduled crawl and miner using it starts failing with invalid_grant … Invalid JWT Signature or "Provided Service account JSON is invalid" - and can go unnoticed for weeks. Put the Atlan credential on your key-rotation runbook: generate the new key, update the credential in Atlan, re-test, then revoke the old key.

Path B, Phase 2 - Workload Identity Federation

Owner: Atlan User + Google Cloud admin

For the full walkthrough, see Set up Workload Identity Federation. In summary:

1. Atlan User - Settings → OAuth Clients → Create OAuth Client. Store the generated Client ID and Client Secret.
2. Google Cloud admin - Create a Workload Identity Pool (e.g. atlan-wif-pool) and an OIDC provider in it (e.g. atlan-oidc-provider) with:
   • Issuer URL: https://<tenant>.atlan.com/auth/realms/default
   • Allowed audience: the OAuth Client ID from step 1
   • Attribute mappings: google.subject → assertion.sub, attribute.audience → assertion.aud
3. Google Cloud admin - Grant the service account from Phase 1 the roles/iam.workloadIdentityUser role for the pool, scoped to attribute audience = your OAuth Client ID.
4. Atlan User - In the crawler choose Workload Identity Federation and enter: Project ID, Service Account Email, WIF Pool Provider ID, OAuth Client ID, OAuth Client Secret.

The WIF Pool Provider ID format trap

The field wants the provider resource path, exactly this shape:

//iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<pool-id>/providers/<provider-id>

A common mistake is pasting the principal string (the one containing /subject/…) instead. If your value has /subject/ in it, it's the wrong string. Note it starts with two slashes and a project number, not the project ID.

Using VPC Service Controls?

WIF's token-exchange calls are evaluated against the WIF principal, not your service account. Your perimeter needs ingress rules written for that principal on sts.googleapis.com and iamcredentials.googleapis.com, or the test fails with GCP STS error: status 403 … prohibited by organization's policy. Full recipe in VPC Service Controls & allowlisting.

Phase 3 - Verify

Owner: Atlan User

• Test connection passes in the crawler setup
• Preflight checks pass (validates metadata permissions, query permission, and - for miners - query history access)
• A crawl scoped to one small dataset completes
• A full crawl and a miner run complete

Preflight fails but you believe the setup is right? It may be a false negative - see Troubleshooting Google BigQuery connectivity before undoing anything.

Next steps

Crawl Google BigQuery: Configure and run the crawler to extract metadata from BigQuery