## Set up Google BigQuery

URL: https://docs.atlan.com/apps/connectors/data-warehouses/google-bigquery/how-tos/set-up-google-bigquery

> You must be a Google BigQuery administrator to run these commands. For more information, see [Google Cloud's Granting, changing, and revoking access to resources](https://cloud.google.com/iam/docs/granting-changing-revoking-access).

:::warning Who can do this?
You must be a Google BigQuery administrator to run these commands. For more information, see [Google Cloud's Granting, changing, and revoking access to resources](https://cloud.google.com/iam/docs/granting-changing-revoking-access).
:::

Atlan [extracts metadata from Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/crawl-google-bigquery/llms.txt) through read-only access. After you crawl metadata for your Google BigQuery assets, you can [mine query history](https://docs.atlan.com/llms/connectors/google-bigquery/mine-google-bigquery/llms.txt) to construct lineage.

If you enable sample data preview or querying, Atlan cost-optimizes previews and queries for [tables](https://docs.atlan.com/llms/connectors/google-bigquery/what-does-atlan-crawl-from-google-bigquery/llms.txt) only. For [views](https://docs.atlan.com/llms/connectors/google-bigquery/what-does-atlan-crawl-from-google-bigquery/llms.txt) and [materialized views](https://docs.atlan.com/llms/connectors/google-bigquery/what-does-atlan-crawl-from-google-bigquery/llms.txt), Atlan shows a cost nudge before you preview or query data. Learn more in [What does Atlan crawl from Google BigQuery?](https://docs.atlan.com/llms/connectors/google-bigquery/what-does-atlan-crawl-from-google-bigquery/llms.txt).

Atlan supports two authentication methods for Google BigQuery:

- **Service account key** (this guide): Use a service account JSON key file for authentication. This is the traditional approach that works in all environments.
- **[Workload Identity Federation](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-workload-identity-federation/llms.txt)**: Use short-lived tokens instead of persistent credentials for enhanced security. Recommended for organizations that want to eliminate long-lived service account keys.

This guide covers service account key authentication. To create a service account, you can use either:

- Google Cloud console
- Google Cloud CLI

Choose one method below. Both options create the same IAM role, service account, and service account key.

## Prerequisites

Before you begin, make sure you have:

- Google BigQuery administrator access to create custom IAM roles, service accounts, and service account keys
- A Google Cloud project where you want to grant Atlan access
- Access to either the Google Cloud console or the Google Cloud CLI (`gcloud`)

## Permissions

Atlan requires the following permissions to extract metadata from Google BigQuery.
Create a custom role with these permissions, then assign the role to the Atlan service account.

### Metadata crawling

Use these permissions for the baseline metadata crawl.
To configure permissions for crawling metadata, add the following permissions to the custom role:

- `bigquery.datasets.get` enables Atlan to retrieve metadata about a dataset.
- `bigquery.datasets.getIamPolicy` enables Atlan to read a dataset's IAM permissions.
- `bigquery.jobs.create` enables Atlan to run jobs (including queries) within the project.

 :::warning
 Without this, Atlan can't query the source.
 :::

- `bigquery.routines.get` enables Atlan to retrieve routine definitions and metadata.
- `bigquery.routines.list` enables Atlan to list routines and metadata on routines.
- `bigquery.tables.get` enables Atlan to retrieve table metadata.
- `bigquery.tables.getIamPolicy` enables Atlan to read a table's IAM policy.
- `bigquery.tables.list` enables Atlan to list tables and metadata on tables.
- `bigquery.readsessions.create` enables Atlan to create a session to stream large results.
- `bigquery.readsessions.getData` enables Atlan to retrieve data from the session.
- `bigquery.readsessions.update` enables Atlan to cancel the session.
- `resourcemanager.projects.get` enables Atlan to retrieve project names and metadata.

Atlan uses the [BigQuery tables.get](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/get) API endpoint to capture metadata. If you crawl external Delta Lake format tables that aren't created as BigLake tables, BigQuery checks the latest Delta Lake checkpoint to detect schema changes. Add these Cloud Storage permissions to the custom role:

- `storage.objects.get`
- `storage.objects.list`

For more information, see [Creating Delta Lake tables](https://docs.cloud.google.com/bigquery/docs/create-delta-lake-table#create-tables).

### Add data preview and querying

Add these permissions if you enable data preview or querying for the connection.
To configure permissions for previewing and querying data, add the following permissions to the custom role:

- `bigquery.tables.getData` enables Atlan to retrieve table data.

 :::warning
 This permission is also required for retrieving metadata such as the row count and update time of a table.
 :::

- `bigquery.jobs.get` enables Atlan to retrieve data and metadata on any job, including queries.
- `bigquery.jobs.listAll` enables Atlan to list all jobs and retrieve metadata on any job submitted by any user.
- `bigquery.jobs.update` enables Atlan to cancel any job, including a running query.

### Add query history mining

Add these permissions if you mine query history to build lineage.
Atlan currently doesn't support generating lineage using the `bq cp` commands - for example, `bq cp <source-table> <destination-table>`.
To configure permissions for mining query history, add the following permissions to the custom role:

- `bigquery.jobs.listAll` enables Atlan to fetch all queries for a project.
- `bigquery.jobs.get` enables Atlan to access query text for queries.

### Crawl tags

Add these permissions if you crawl tags or policy tags from Google BigQuery.
To configure permissions for crawling [Google BigQuery tags and policy tags](https://docs.atlan.com/llms/connectors/google-bigquery/manage-google-bigquery-tags/llms.txt), add the following permissions to the custom role:

- `resourcemanager.tagKeys.list` enables Atlan to fetch all tag keys.
- `resourcemanager.tagValues.list` enables Atlan to fetch all tag values for tag keys.
- `datacatalog.taxonomies.list` enables Atlan to fetch all policy tag taxonomies.
- `datacatalog.taxonomies.get` enables Atlan to fetch all policy tag taxonomies.

### Add data quality

If you plan to use [Data Quality Studio](https://docs.atlan.com/llms/governance/data-quality/bigquery/llms.txt) with BigQuery, additional setup is required - including a dedicated service account, dataset, and IAM roles. See [Set up BigQuery for Data Quality](https://docs.atlan.com/llms/governance/data-quality/set-up-bigquery/llms.txt) for the full guide.

## Create custom role and service account

Create a custom role, service account, and service account key for Atlan. You can use the Google Cloud console or Google Cloud CLI.

### Google Cloud console

### Create custom role

[Create a custom role](https://cloud.google.com/iam/docs/creating-custom-roles#iam-custom-roles-create-console) in the Google Cloud console.

1. Open the [Google Cloud console](https://console.cloud.google.com).
2. From the left menu under _IAM and admin_, click **Roles**.
3. Using the dropdown list at the top of the page, select the project in which you want to create a role.
4. From the upper left of the _Roles_ page, click **Create Role**.
5. In the _Create role_ page, enter the following details:
 1. For _Title_, enter a meaningful name for the custom role - for example, `Atlan User Role`.
 2. For _Description_, enter a description for the custom role if needed.
 3. For _ID_, the Google Cloud console generates a custom role ID based on the custom role name. Edit the ID if necessary - the ID can't be changed later.
 4. For _Role launch stage_, assign a stage if needed - for example, _Alpha_ or _General availability_.
 5. Click **Add permissions** to select the permissions you want to include in the custom role. In the _Add permissions_ dialog, click the **Enter property name or value** filter and add the required and any optional permissions.
 6. Click **Create** to finish custom role setup.

### Create service account

Create a service account and add the custom role to it.

1. Open the [Google Cloud console](https://console.cloud.google.com).
2. From the left menu under _IAM and admin_, click **Service accounts**.
3. Select a Google Cloud project.
4. From the upper left of the _Service accounts_ page, click **Create Service Account**.
5. For _Service account details_, enter the following details:
 1. For _Service account name_, enter a service account name to display in the Google Cloud console.
 2. For _Service account ID_, the Google Cloud console generates a service account ID based on this name. Edit the ID if necessary - the ID can't be changed later.
 3. For _Service account description_, enter a description for the service account if needed.
 4. Click **Create and continue** to proceed to the next step.
6. For _Grant this service account access to the project_, enter the following details:
 1. Click the **Select a role** dropdown and then select the custom role you created earlier - for example, `Atlan User Role`.
 2. Click **Continue** to proceed to the next step.
7. Click **Done** to finish the service account setup.

### Create service account key

Create a service account key for [crawling Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/crawl-google-bigquery/llms.txt).

1. Open the [Google Cloud console](https://console.cloud.google.com).
2. From the left menu under _IAM and admin_, click **Service accounts**.
3. Select the Google Cloud project for which you created the service account.
4. On the _Service accounts_ page, click the email address of the service account that you want to create a key for.
5. From the upper left of your service account page, click the **Keys** tab.
6. On the _Keys_ page, click the **Add Key** dropdown and then click **Create new key**.
7. In the _Create private key_ dialog, for _Key type_, click **JSON** and then click **Create**. This creates a service account key file. Download the key file and store it in a secure location. You can't download it again.

### Google Cloud CLI

### Prerequisites

Set up the Google Cloud CLI in any one of the following development environments:

- **Cloud Shell** - to use an online terminal with the gcloud CLI already set up, activate Cloud Shell:
 - To launch a Cloud Shell session from the Google Cloud console, open the [Google Cloud console](https://console.cloud.google.com), and from the top right, click the **Activate Cloud Shell** icon.
 - A Cloud Shell session starts and displays a command-line prompt. It can take a few seconds for the session to initialize.
- **Local shell** - to use a local development environment, [install](https://cloud.google.com/sdk/docs/install) and [initialize](https://cloud.google.com/sdk/docs/initializing) the gcloud CLI.

### Create custom role

To create a custom role with the requisite and any optional permissions, run the following command:

```shell
gcloud iam roles create atlanUserRole --project=<project_id> \
 --title="Atlan User Role" --description="Atlan User Role to extract metadata" \
 --permissions="bigquery.datasets.get,bigquery.datasets.getIamPolicy,bigquery.jobs.create,bigquery.readsessions.create,bigquery.readsessions.getData,bigquery.readsessions.update,bigquery.routines.get,bigquery.routines.list,bigquery.tables.get,bigquery.tables.getIamPolicy,bigquery.tables.list,resourcemanager.projects.get" \
 --stage=ALPHA
```

- Replace `<project_id>` with the project ID of your Google Cloud project.

### Create service account

To create a service account, run the following command:

```shell
gcloud iam service-accounts create atlanUser \
 --description="Atlan Service Account to extract metadata" \
 --display-name="Atlan User"
```

To add your custom role to your service account, run the following command:

```shell
gcloud projects add-iam-policy-binding <project_id> \
 --member="serviceAccount:atlanUser@<project_id>.iam.gserviceaccount.com" \
 --role="atlanUserRole"
```

- Replace `<project_id>` with the project ID of your Google Cloud project.

### Create service account key

To create a service account key, run the following command:

```shell
gcloud iam service-accounts keys create <key_file_path> \
 --iam-account="atlanUser@<project_id>.iam.gserviceaccount.com"
```

- Replace `<key_file_path>` with a path to a new output file for the private key - for example, `~/atlanUser-private-key.json`.
- Replace `<project_id>` with the project ID of your Google Cloud project.

## Troubleshooting

If you run into permission or authentication issues, see [Troubleshoot Google BigQuery connectivity](https://docs.atlan.com/llms/connectors/google-bigquery/troubleshooting-google-bigquery-connectivity/llms.txt). If you still need help, contact Atlan support.

## Next steps

* [Crawl Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/crawl-google-bigquery/llms.txt): Create a connection and run the crawler to extract metadata from Google BigQuery
* [How Atlan connects to Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/how-atlan-connects-to-google-bigquery/llms.txt): Connectivity, authentication, and Self-deployed runtime options
* [Set up BigQuery for Data Quality](https://docs.atlan.com/llms/governance/data-quality/set-up-bigquery/llms.txt): Configure BigQuery for Data Quality Studio monitoring

---