## Mine Snowflake

URL: https://docs.atlan.com/apps/connectors/data-warehouses/snowflake/how-tos/mine-snowflake

> Once you have [crawled assets from Snowflake](/apps/connectors/data-warehouses/snowflake/how-tos/crawl-snowflake), you can mine its query history to construct lineage.

Atlan mines Snowflake query history to construct column-level lineage across your warehouse.

Once you have [crawled assets from Snowflake](https://docs.atlan.com/llms/connectors/snowflake/crawl-snowflake/llms.txt), you can mine its query history to construct lineage.

To mine lineage from Snowflake, review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) and then complete the following steps.

## Select the miner

To select the Snowflake miner:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Snowflake Miner** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

## Configure miner

To configure the Snowflake miner:

1. For _Connection_, select the connection to mine. (To select a connection, [the crawler](https://docs.atlan.com/llms/connectors/snowflake/crawl-snowflake/llms.txt) must have already run.)

2. For _Miner Extraction Method_, select **Source**, **Agent**, or see the separate instructions for [Generic Miner](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt).

3. For _Snowflake Database_:
 - If the connection is configured with [access to the snowflake database](https://docs.atlan.com/llms/connectors/snowflake/set-up-snowflake/llms.txt), choose **Default**.
 - If the connection can only [access a separate cloned database](https://docs.atlan.com/llms/connectors/snowflake/set-up-snowflake/llms.txt), choose **Cloned Database**.
4. If you are using a cloned database, enter the name of the cloned database in _Database Name_ and the name of the cloned schema in _Schema Name_.

5. For _Start time_, choose the earliest date from which to mine query history.

 :::info

 💪 **Did you know?** The miner restricts you to only querying the past two weeks of query history. If you need to query more history, for example in an initial load, consider using [Generic Miner](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt) first. After the initial load, you can [modify the miner's configuration](https://docs.atlan.com/llms/catalog/connector-capabilities/manage-connectivity/llms.txt) to use query history extraction.

 :::

6. (Optional) To replace literal values in mined query history with placeholders, change _Redact SQL Text_ to **True**. Atlan preserves query structure, table references, and column references, so lineage is unaffected. This option is off by default, applies to future runs only, and increases miner runtime. For what Atlan redacts and the limitations to plan for, see [Query redaction for Snowflake](https://docs.atlan.com/llms/connectors/snowflake/query-redaction/llms.txt).

7. To check for any permissions or other configuration issues before running the miner, click **Preflight checks**.

8. At the bottom of the screen, click **Next** to proceed.

### Agent extraction method

Atlan supports using a Secure Agent for mining query history from Snowflake. To use a Secure Agent, follow these steps:

:::info Important!
Query redaction is a compute-intensive operation and can impact workflow performance. To use this feature effectively, assign at least 4 CPU cores to the agent.
:::

1. Select the **Agent** tab.

2. **Configure Query Redaction**: By default, Query Redaction is turned off. You may choose to configure query redaction:
 - Set `Enable Query Redaction` flag to true if you want to redact PII data from query history before ingesting it to Atlan. 
 - **Configure PII Patterns for Query Redaction**: Define patterns to identify and redact sensitive information using the following JSON format:
 ```json
 {
 "pii_patterns": [
 {
 "name": "email",
 "regex": "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}\\b",
 "replacement": "[EMAIL]"
 }
 ]
 }
 ```

 :::info Two redaction options, used independently

 `Enable Query Redaction` on this tab is separate from the _Redact SQL Text_ option on the [_Configure miner_ screen](#configure-miner). This one runs on your Secure Agent and matches the regex patterns you define, so it's suited to specific PII formats such as email addresses. _Redact SQL Text_ parses each statement and replaces literal values generically, and works on both direct and Secure Agent connections. You can enable either or both. See [Query redaction for Snowflake](https://docs.atlan.com/llms/connectors/snowflake/query-redaction/llms.txt).

 :::

3. Configure the Snowflake data source by adding the secret keys for your secret store. For details on the required fields, refer to the connection configuration used when [crawling Snowflake](https://docs.atlan.com/llms/connectors/snowflake/crawl-snowflake/llms.txt).

4. Complete the Secure Agent configuration by following the instructions in the [How to configure Secure Agent for workflow execution](https://docs.atlan.com/llms/platform/secure-agent/configure-secure-agent-for-workflow-execution/llms.txt) guide.

5. Click **Next** after completing the configuration.

:::warning Important!
If running the miner for the first time, Atlan recommends setting a start date around three days prior to the current date and then scheduling it daily to build up to two weeks of query history. Mining two weeks of query history on the first miner run may cause delays. For all subsequent runs, Atlan requires a minimum lag of 24 to 48 hours to capture all the relevant transformations that were part of a session. Learn more about the miner logic [here](https://docs.atlan.com/llms/catalog/lineage/troubleshooting-lineage/llms.txt).
:::

## Configure miner behavior

To configure the Snowflake miner behavior:

1. (Optional) For _Calculate popularity_, keep **True** to retrieve [usage and popularity metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) for your Snowflake assets from query history.
 - For _Excluded Users_, type the names of users to be excluded while calculating [usage metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/interpret-usage-metrics/llms.txt) for Snowflake assets. Press `Enter` after each name to add more names. 
2. (Optional) For _Advanced Config_, keep _Default_ for the default configuration or click **Custom** to configure the miner:
 - If Atlan support has provided you with a custom control configuration, enter the configuration into the _Custom Config_ box.
 - You can also enter `{“ignore-all-case”: true}` to enable crawling assets with case-sensitive identifiers.
 - For _Popularity Window (days)_, 90 days is the maximum limit. You can set a shorter popularity window of less than 90 days.

## Run the miner

To run the Snowflake miner, after completing the configuration steps:

- To run the miner once immediately, at the bottom of the screen, click the **Run** button.
- To schedule the miner to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule & Run** button.

Once the miner has completed running, you can see lineage for Snowflake assets that were created in Snowflake between the start time and when the miner ran! 🎉

:::note
Snowflake truncates queries longer than 100K characters in its [query history](https://docs.snowflake.com/en/sql-reference/account-usage/query_history). Atlan can't build lineage from a truncated query, so lineage may be missing for the assets they created.
:::

---