Skip to main content

Special cases

TL;DR

Google BigQuery special cases - multiple projects, Delta external tables, Data Quality with WIF, per-user SSO, credential rotation, and the Lakehouse integration.

Connect

Multiple projects, cross-project crawling, Delta external tables, Data Quality with WIF, per-user SSO, credential rotation, and the Lakehouse integration.

Crawling many GCP projects with one connection​

One BigQuery connection can crawl every project its service account can see. Three rules keep it healthy:

  • Bind the role everywhere. The Atlan custom role must be bound in the service account's home project (where the account was created) and every target project. Atlan's test connection validates permissions against the home project - a common failure for SDK-created connections crawling only other projects.
  • Mind your include/exclude filters. If the filter regex matches zero of the projects the account can see, the crawl fails with a generic "unable to connect" error that looks like an auth problem - and can keep failing unnoticed for weeks on a schedule. When a crawl breaks right after a filter change, suspect the filter first.
  • VPC-SC users: allowlist each new project before adding it - see VPC Service Controls & allowlisting.

External tables in Delta format​

For Delta-format external tables, BigQuery itself reads transaction logs from Cloud Storage during metadata calls. The crawl then needs storage.objects.get and storage.objects.list (already in the optional permission set in Set up authentication) - and, under VPC-SC, permission for that storage path. Symptom when missing: crawls fail only on Delta tables with "Permission denied while globbing file pattern" or a does not have storage.objects.get access error naming a gs://…/_delta_log/… path. Other external formats (e.g. Avro) don't trigger this.

Data Quality studio + WIF​

The Data Quality setup pipeline currently requires a service account JSON key and fails when a WIF credential is selected (the UI error is an unhelpful "Something went wrong while configuring connection settings."). The working pattern until WIF support ships:

  • Keep WIF for crawling/mining.
  • Create a separate, dedicated service account with a JSON key just for Data Quality, granting additionally: bigquery.datasets.create, bigquery.tables.create, bigquery.tables.updateData, bigquery.jobs.create.
  • If your policy bans key files entirely, raise it with your account team - WIF-for-DQ is roadmapped.

Rotating credentials without breaking schedules​

  • UI: edit the connection's credential, paste the new key JSON (or new WIF values), and use Test connection before revoking the old key.
  • API: update via POST /api/service/credentials/<credential-guid> - the GUID is in your workflow's config (GET /api/service/workflows/<workflow-id>, parameter credential-guid).
/credentials/test does not save anything

POST /api/service/credentials/test only validates a credential - it returns 200 and persists nothing. An automation can rotate keys against it for weeks, convinced it's working. If you rotate programmatically, verify by reading the credential back or running the workflow.

When replacing the service account under WIF, the new account also needs roles/iam.workloadIdentityUser on the pool - permissions alone aren't enough.

Per-user SSO (OAuth) connections​

Connections using per-user Google OAuth (each analyst queries as themselves) depend on your OAuth app configuration, not Atlan's:

  • The OAuth consent screen must be set to Internal (or the app published - BigQuery scopes trigger Google's verification process). An app left in Testing mode only allows its listed test users; everyone else gets "Account restricted" / login failures.
  • Organization zero-trust/device policies can block the browser redirect step; test from a managed browser profile in incognito.
  • Fallback: switch the connection to service-account auth - losing per-user permission enforcement.

Lakehouse (Iceberg) integration - traffic in the other direction​

With the Atlan Lakehouse integration, your BigQuery reaches out to Atlan - the reverse of crawling. Two things to prepare:

  • VPC-SC egress rule allowing your BigQuery service identities (including Google's managed BigQuery connection delegate account, bqcx-…@gcp-sa-bigquery-condel.iam.gserviceaccount.com) to call storage.googleapis.com (objects.get, objects.list, buckets.get) against Atlan's GCP project.
  • HTTPS egress to your tenant's API (https://tenant.atlan.com/api/polaris, port 443). No PSC is required for this path.

Self-deployed runtime (SDR)​

If policy requires that credentials never leave your network, run the extraction inside it: the self-deployed runtime calls the BigQuery API locally, reads credentials from your own secret store, and pushes results to Atlan over outbound HTTPS (port 443) - no inbound connectivity needed. Both auth methods work. One field note: on SDR + WIF connections there is no key file anywhere, so any error message blaming a "Service account JSON" is misattributed - see Troubleshooting Google BigQuery connectivity.