## Crawl Google BigQuery URL: https://docs.atlan.com/apps/connectors/data-warehouses/google-bigquery/how-tos/crawl-google-bigquery > Crawl metadata from Google BigQuery to catalog datasets, tables, views, and columns. Extract lineage and asset details after completing prerequisite setup. Extract metadata from your [Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-google-bigquery/llms.txt) data warehouse into Atlan to catalog datasets, tables, views, and enable data discovery. After completing the prerequisite setup, you can crawl to extract lineage and column-level details. Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting. ## Prerequisites Before you begin, verify you have: * Completed the [Set up Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-google-bigquery/llms.txt) guide or [Set up Workload Identity Federation](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-workload-identity-federation/llms.txt) * Access to your Google Cloud project and BigQuery * Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) ## Create crawler workflow Create a new workflow and select Google BigQuery as your connector source. 1. In your Atlan workspace, click **Connectors** in the left sidebar. * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation. 2. Click **Marketplace**. 3. Search for **BigQuery Assets** and select it. 4. Click **Install**. 5. Once installation completes, click **Setup Workflow** on the same tile. ### Configure extraction When setting up metadata extraction from your Google BigQuery data, choose how Atlan connects and extracts metadata. Select the extraction method that best fits your organization's security and network requirements: ### Direct Atlan SaaS connects directly to Google BigQuery (typically via the public BigQuery API or Private Service Connect). This method supports multiple authentication options and lets you test the connection before proceeding. 1. For **Connectivity**, choose how you want Atlan to connect to Google BigQuery: - **Public Network**: Connect using the public BigQuery API endpoint from Google. - **Private Network Link**: Connect through a private endpoint. [Contact Atlan support](https://docs.atlan.com/support/submit-request) to request the DNS name of the Private Service Connect endpoint. For **Host**, enter the DNS name in the format `https://bigquery-.p.googleapis.com`. Replace `` with the DNS name. For **Port**, `443` is the default. 2. Choose an authentication method for your direct connection. 1. Use a [service account key](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-google-bigquery/llms.txt) for authentication. - **Project Id**: Enter the value of `project_id` from the JSON for the service account you created. This project ID is used to authenticate the connection. You can configure the crawler to extract more than the specified project. - **Service Account Json**: Paste the entire JSON for the service account you created. - **Service Account Email**: Enter the value of `client_email` from the JSON for the service account you created. 2. After entering the authentication details, click **Test Authentication** to verify your configuration. If the test is successful, click **Next** to proceed with the connection configuration. ### Workload Identity Federation 1. Authenticate using [Workload Identity Federation](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-workload-identity-federation/llms.txt). - **Project Id**: Enter your Google Cloud project ID. This project ID is used to authenticate the connection. You can configure the crawler to extract more than the specified project. - **Service Account Email**: Enter the email of the service account that has BigQuery permissions and is configured for WIF impersonation. - **WIF Pool Provider Id**: Enter the full resource name of your WIF provider in the following format: ``` //iam.googleapis.com/projects//locations/global/workloadIdentityPools//providers/ ``` - **Atlan OAuth Client Id**: Enter the OAuth Client ID you created in Atlan during [WIF setup](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-workload-identity-federation/llms.txt). - **Atlan OAuth Client Secret**: Enter the OAuth Client Secret you created in Atlan. 2. After entering the authentication details, click **Test Authentication** to verify your configuration. If the test is successful, click **Next** to proceed with the connection configuration. ### Agent Atlan's Secure Agent application is deployed within your organization and connects to Google BigQuery. This method provides additional security by keeping connections within your network perimeter. 1. Install Self-Deployed Runtime if you haven't already: - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt) - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt) 2. For **Connectivity**, choose how you want Atlan to connect to Google BigQuery: - **Public Network**: Connect using the public BigQuery API endpoint from Google. - **Private Network Link**: Connect through a private endpoint. [Contact Atlan support](https://docs.atlan.com/support/submit-request) to request the DNS name of the Private Service Connect endpoint. For **Host**, enter the DNS name in the format `https://bigquery-.p.googleapis.com`. Replace `` with the DNS name. For **Port**, `443` is the default. 3. Choose an authentication method for your agent-based connection. 1. Use a [service account key](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-google-bigquery/llms.txt) for authentication. - **Project Id**: Enter the value of `project_id` from the JSON for the service account you created. - **Service Account Json**: Enter the secret key name for the service account JSON. For format requirements, see [Configure workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt). - **Service Account Email**: Enter the value of `client_email` from the JSON for the service account you created. 2. Click **Next** to proceed with the connection configuration. ### Workload Identity Federation 1. Authenticate using [Workload Identity Federation](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-workload-identity-federation/llms.txt) from within your agent environment. - **Project Id**: Enter your Google Cloud project ID. - **Service Account Email**: Enter the email of the service account that has BigQuery permissions and is configured for WIF impersonation. - **WIF Pool Provider Id**: Enter the full resource name of your WIF provider in the following format: ``` //iam.googleapis.com/projects//locations/global/workloadIdentityPools//providers/ ``` - **Atlan OAuth Client Id**: Enter the OAuth Client ID you created in Atlan during WIF setup. - **Atlan OAuth Client Secret**: Enter the secret key name for the OAuth Client Secret. For format requirements, see [Configure workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt). 2. Click **Next** to proceed with the connection configuration. ### Configure connection Set up the connection name and access controls for your Google BigQuery data source in Atlan. 1. Provide a **Connection Name** that represents your source environment. For example, you might use values like `production`, `development`, `gold`, or `analytics`. 2. To change the users able to manage this connection, update the users or groups listed under **Connection Admins**. If you don't specify any user or group, nobody can manage the connection (not even admins). 3. To prevent users from querying Google BigQuery data, set **Allow SQL Query** to **No**. 4. To prevent users from previewing Google BigQuery data, set **Allow Data Preview** to **No**. 5. At the bottom of the screen, click **Next** to proceed. ### Configure crawler Before running the crawler, you can configure which assets to include or exclude and other crawler options. Include and exclude metadata filters are only available when using the direct extraction method. If an asset appears in both the include and exclude filters, the exclude filter takes precedence. - For **Filter Sharded Tables**, keep **No** for the default configuration or click **Yes** to enable Atlan to catalog and display [sharded tables](https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard) with the same naming prefix as a single table in asset discovery and the lineage graph. - To exclude specific assets from crawling, select **Exclude Metadata**. This defaults to no assets if none are specified. - To include specific assets in crawling, select **Include Metadata**. This defaults to all assets if none are specified. - To ignore tables and views based on a naming convention, specify a regular expression in the **Exclude regex for tables & views** field. - To [import existing tags from Google BigQuery to Atlan](https://docs.atlan.com/llms/connectors/google-bigquery/manage-google-bigquery-tags/llms.txt), for **Import Tags**, click **Yes**. - For **Import Nested Columns**, keep **Yes** (the default) to extract metadata for nested columns in `RECORD` (STRUCT) and `REPEATED` (ARRAY) fields. When enabled, Atlan crawls nested columns up to BigQuery's [maximum nesting depth of 15 levels](https://docs.atlan.com/llms/connectors/google-bigquery/troubleshooting-google-bigquery-connectivity/llms.txt). Click **No** to skip nested column extraction and crawl only top-level columns. - For **Advanced Config**, keep **Default** for the default configuration or click **Custom** if Atlan support has provided you with a custom control configuration: - Enter the configuration into the **Custom Config** box. You can also enter `{"ignore-all-case": true}` to enable crawling assets with case-sensitive identifiers. - For **Hidden Assets**, keep **No** for the default configuration or click **Yes** to crawl metadata from your [hidden datasets](https://cloud.google.com/bigquery/docs/datasets#hidden_datasets) in Google BigQuery. ### Run crawler ### Direct 1. Click **Preflight checks** to validate permissions and configuration before running the crawler. This helps identify any [potential issues](https://docs.atlan.com/llms/connectors/google-bigquery/preflight-checks-for-google-bigquery/llms.txt) early. 2. After the preflight checks pass, you can either: - Click **Run** to run the crawler once immediately. - Click **Schedule Run** to schedule the crawler to run hourly, daily, weekly, or monthly. ### Agent You can either: - Click **Run** to run the crawler once immediately. - Click **Schedule Run** to schedule the crawler to run hourly, daily, weekly, or monthly. Once the crawler has completed running, you can see the assets in Atlan's asset page. ## See also * [How Atlan connects to Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/how-atlan-connects-to-google-bigquery/llms.txt): Connectivity, authentication, and data access patterns * [What does Atlan crawl from Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/what-does-atlan-crawl-from-google-bigquery/llms.txt): Learn about the Google BigQuery assets and metadata that Atlan discovers and catalogs * [Preflight checks for Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/preflight-checks-for-google-bigquery/llms.txt): Verify permissions and configuration before crawling * [Troubleshooting Google BigQuery connectivity](https://docs.atlan.com/llms/connectors/google-bigquery/troubleshooting-google-bigquery-connectivity/llms.txt): Resolve common connection issues ---