## Mine Google BigQuery

URL: https://docs.atlan.com/apps/connectors/data-warehouses/google-bigquery/how-tos/mine-google-bigquery

> Once you have [crawled assets from Google BigQuery](/apps/connectors/data-warehouses/google-bigquery/how-tos/crawl-google-bigquery), you can mine its query history to construct lineage.

Once you have [crawled assets from Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/crawl-google-bigquery/llms.txt), you can mine its query history to construct lineage. The miner supports both Direct and Agent extraction methods.

To mine lineage from Google BigQuery, review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) and then complete the following steps.

## Select miner

To select the Google BigQuery miner:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **BigQuery Miner** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

## Configure miner

To configure the Google BigQuery miner:

1. For _Connection_, select the connection to mine. (To select a connection, [the crawler](https://docs.atlan.com/llms/connectors/google-bigquery/crawl-google-bigquery/llms.txt) must have already run.)

2. For _Miner Extraction Method_, select **Query History**, **Offline**, or **Agent**.

3. For _Start time_, choose the earliest date from which to mine query history.

 :::info

 💪 **Did you know?** The miner restricts you to only querying the past two weeks of query history. If you need to query more history, for example in an initial load, consider using [Generic Miner](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt) first. After the initial load, you can [modify the miner's configuration](https://docs.atlan.com/llms/catalog/connector-capabilities/manage-connectivity/llms.txt) to use query history extraction.

 :::

4. (Optional) By default, the miner fetches data from the default region (United States). To fetch data from [another region](https://cloud.google.com/bigquery/docs/locations), for _Region_, select **Custom** and then enter the region where your `INFORMATION_SCHEMA` is hosted under _Custom BigQuery Region_. Enter the region in the [following format](https://cloud.google.com/bigquery/docs/information-schema-intro#region_qualifier) `region-`, replacing `` with your specific region - for example, `europe-north1`.

5. To check for any permissions or other configuration issues before running the miner, click **Preflight checks**.

6. At the bottom of the screen, click **Next** to proceed.

:::warning

If running the miner for the first time, Atlan recommends setting a start date roughly three days prior to the current date and then scheduling it daily to build up to two weeks of query history. Mining two weeks of query history on the first miner run may cause delays. Atlan requires a minimum lag of 24 to 48 hours to capture all the relevant transformations that were part of a session. Learn more about the miner logic [here](https://docs.atlan.com/llms/catalog/lineage/troubleshooting-lineage/llms.txt).

:::

### Configure agent extraction

If your organization requires connections to remain within your network perimeter, use the Agent extraction method instead of Direct. To use a Secure Agent, follow these steps:

1. Select the **Agent** tab.

2. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
3. For **Connectivity**, choose how you want Atlan to connect to Google BigQuery:
 - **Public Network**: Connect using the public BigQuery API endpoint from Google.
 - **Private Network Link**: Connect through a private endpoint. [Contact Atlan support](https://docs.atlan.com/support/submit-request) to request the DNS name of the Private Service Connect endpoint. For **Host**, enter the DNS name in the format `https://bigquery-<privateserver>.p.googleapis.com`. Replace `<privateserver>` with the DNS name. For **Port**, `443` is the default.
4. Choose an authentication method for your agent-based connection and configure the data source by adding the secret keys for your secret store:

### Service account

- **Project Id**: Enter the secret key name for `project_id`.
 - **Secret Key for Service Account JSON**: Enter the secret key name for the service account JSON. For format requirements, see [Configure workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).
 - **Secret Key for Service Account Email**: Enter the secret key name for `client_email`.

### Workload Identity Federation

- **Project Id**: Enter the secret key name for your Google Cloud project ID.
 - **Secret Key for Service Account Email**: Enter the secret key name for the service account email (used for WIF impersonation).
 - **Secret Key for WIF Pool Provider Id**: Enter the secret key name for the WIF provider resource name.
 - **Secret Key for Atlan OAuth Client Id**: Enter the secret key name for the OAuth Client ID created during [WIF setup](https://docs.atlan.com/llms/connectors/google-bigquery/set-up-workload-identity-federation/llms.txt).
 - **Secret Key for Atlan OAuth Client Secret**: Enter the secret key name for the OAuth Client Secret. For format requirements, see [Configure workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).

5. Complete the Secure Agent configuration by selecting your secret store and entering the secret path. For details, see [Configure workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).

6. Click **Next** after completing the configuration.

## Configure miner behavior

To configure the Google BigQuery miner behavior:

1. (Optional) For _Calculate popularity_, change to **True** to retrieve [usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) for your Google BigQuery assets from query history:
 1. To select a [pricing model for running queries](https://cloud.google.com/bigquery/pricing), for _Pricing Model_, click **On Demand** to be charged for the number of bytes processed or **Flat Rate** for the number of slots purchased.
 2. For _Popularity Window (days)_, 30 days is the maximum limit. You can set a shorter popularity window of less than 30 days.
 3. For _Excluded Users_, type the names of users to be excluded while calculating [usage metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) for Google BigQuery assets. Press `enter` after each name to add more names.
2. (Optional) For _Control Config_, click **Custom** to configure the following:
 1. For _Fetch excluded project's QUERY\_HISTORY_, click **Yes** to mine query history from databases or projects excluded while [crawling metadata from Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/crawl-google-bigquery/llms.txt).
 2. If Atlan support has provided you with a custom control configuration, enter the configuration into the _Custom Config_ box. You can also:
 - (Optional) Enter `{“ignore-all-case”: true}` to enable crawling assets with case-sensitive identifiers.

## Run miner

To run the Google BigQuery miner, after completing the previous steps:

- To run the miner once immediately, at the bottom of the screen, click the **Run** button.
- To schedule the miner to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule & Run** button.

Once the miner has completed running, you can see lineage for Google BigQuery assets that were created in Google BigQuery between the start time and when the miner ran.

---