Skip to main content

How Atlan connects to Google BigQuery

Atlan connects to your Google BigQuery data warehouse to extract technical metadata while maintaining network security and compliance. You can choose Direct connectivity when BigQuery is available from the internet or Self-deployed runtime when connections must stay within your network perimeter.

Connect via direct network connection​

Atlan's Google BigQuery workflow establishes a direct connection to the BigQuery API from the Atlan SaaS tenant. Use this approach when the BigQuery API is available from the internet (public endpoint or Private Service Connect).

  • Atlan's BigQuery workflow connects to the BigQuery API over HTTPS (port 443) from the Atlan SaaS tenant.
  • You authenticate using a service account key or Workload Identity Federation. Atlan uses these credentials to call BigQuery APIs for metadata only.
  • For private connectivity, you can use Private Service Connect and provide the DNS name Atlan gives you so that traffic stays on your chosen network path.

For details on how direct connectivity works, see Direct connectivity.

Connect via self-deployed runtime​

A runtime service deployed within your organization acts as a secure bridge between Atlan Cloud and Google BigQuery. This approach is required only when you want BigQuery API calls to originate from your network (for example, to satisfy egress or VPC requirements).

  • The runtime maintains an outbound HTTPS connection to Atlan Cloud (port 443) and uses your environment's network to call the BigQuery API over HTTPS (port 443).
  • When you create a crawler workflow, Atlan Cloud sends metadata extraction requests to the runtime. The runtime authenticates to Google Cloud (using a service account key or Workload Identity Federation from your secret store or GKE) and calls the BigQuery API, then returns metadata to Atlan Cloud.
  • Your BigQuery project is only accessed from within your network; Atlan Cloud never receives your credentials when using Self-deployed runtime.

For details on how Self-Deployed Runtime works, see SDR connectivity.

Authenticate with workload identity federation​

When you use Workload Identity Federation, Atlan accesses BigQuery with short-lived credentials instead of a service account key. The same authentication flow applies whether you use Direct connectivity or Self-deployed runtime; only the component that runs the connector (Atlan SaaS or the runtime in your network) differs. The following diagram shows the authentication-specific event flow.

  • One-time setup: You create an OAuth client in Atlan (client ID and secret) and a workload identity pool in Google Cloud with an OIDC provider that trusts your Atlan tenant's issuer. You configure attribute mapping and grant the pool permission to impersonate a Google Cloud service account that has BigQuery access. No service account key is stored.

  • At crawl time: The Atlan connector (SaaS or runtime) requests an OIDC token from Atlan's identity using the OAuth client credentials. It sends this token to Google Workload Identity Federation. WIF validates the token and returns short-lived Google Cloud credentials for the impersonated service account. The connector then calls the BigQuery API with those credentials. Tokens and short-lived credentials aren't stored long-term.

This flow keeps long-lived keys out of the connector; only the OAuth client secret (in Atlan) and the WIF configuration (in GCP) are required for ongoing access.

Authentication and data access​

  • Authentication: Atlan supports service account keys and Workload Identity Federation.

  • Metadata only: Atlan uses BigQuery APIs to read metadata (projects, datasets, tables, views, columns, routines). For data preview or querying, Atlan runs read-only jobs; the permissions you grant to the service account control what Atlan can access.

  • Credential handling: In Direct connectivity, Atlan encrypts credentials at rest and in transit. In Self-deployed runtime, credentials stay in your perimeter; the runtime reads them from your secret store (for example, GCP Secret Manager or HashiCorp Vault) and Atlan Cloud never stores them.

See also​