## Crawl Snowflake URL: https://docs.atlan.com/apps/connectors/data-warehouses/snowflake/how-tos/crawl-snowflake > To crawl metadata from Snowflake, review the [order of operations](/product/connections/how-tos/order-workflows) and then complete the following steps. Atlan crawls Snowflake to catalog databases, schemas, tables, and views, then builds column-level lineage from query history. Once you have configured the [Snowflake user permissions](https://docs.atlan.com/llms/connectors/snowflake/set-up-snowflake/llms.txt), you can establish a connection between Atlan and Snowflake. If you also use [AWS PrivateLink](https://docs.atlan.com/llms/connectors/snowflake/set-up-an-aws-private-network-link-to-snowflake/llms.txt) or [Azure Private Link](https://docs.atlan.com/llms/connectors/snowflake/set-up-an-azure-private-network-link-to-snowflake/llms.txt) for Snowflake, complete that setup first. To crawl metadata from Snowflake, review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) and then complete the following steps. ## Select source To select Snowflake as your source: 1. In your Atlan workspace, click **Connectors** in the left sidebar. * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation. 2. Click **Marketplace**. 3. Search for **Snowflake Assets** and select it. 4. Click **Install**. 5. Once installation completes, click **Setup Workflow** on the same tile. ## Provide credentials Choose your extraction method: :::warning Offline extraction sunset The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt). ::: - In **Direct** extraction, Atlan connects to your database and crawls metadata directly. - In **Agent** extraction, Atlan's secure agent executes metadata extraction within the organization's environment. ### Direct extraction method To enter your Snowflake credentials: 1. For _Account Identifiers (Host)_, enter the hostname, [AWS PrivateLink endpoint](https://docs.atlan.com/llms/connectors/snowflake/set-up-an-aws-private-network-link-to-snowflake/llms.txt), or [Azure Private Link endpoint](https://docs.atlan.com/llms/connectors/snowflake/set-up-an-azure-private-network-link-to-snowflake/llms.txt) for your Snowflake instance. 2. For _Authentication_, choose the method you configured when [setting up the Snowflake user](https://docs.atlan.com/llms/connectors/snowflake/set-up-snowflake/llms.txt): - For **Basic** authentication, enter the _Username_ and _Password_ you configured in either Snowflake or the identity provider. :::info 💪 **Did you know?** Snowflake recommends transitioning away from basic authentication using username and password. Change to [key-pair authentication](https://docs.atlan.com/llms/connectors/snowflake/set-up-snowflake/llms.txt) for enhanced security. For any existing Snowflake workflows, you can [modify the crawler configuration](https://docs.atlan.com/llms/catalog/connector-capabilities/manage-connectivity/llms.txt) to update the authentication method. ::: - For **Keypair** authentication, enter the _Username_, _Encrypted Private Key_, and _Private Key_ _Password_ you configured. Atlan only supports encrypted private keys with a non-empty passphrase, which Snowflake recommends for stronger security. An empty passphrase results in workflow failures. To generate an encrypted private key, refer to [Snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth). - For **Okta SSO** authentication, enter the _Username_, _Password_, and _Authenticator_ you configured. The _Authenticator_ is the [Okta URL endpoint of your Okta account](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use#native-sso-okta-only), typically in the form of `https://.okta.com`. 3. For _Role_, select the Snowflake role that runs the crawler. 4. For _Warehouse_, select the Snowflake warehouse where the crawler runs. 5. Click **Test Authentication** to confirm connectivity to Snowflake using these details. 6. Once successful, at the bottom of the screen, click **Next**. ## Configure connection To complete the Snowflake connection configuration: 1. Provide a _Connection Name_ that represents your source environment. For example, you might use values like `production`, `development`, `gold`, or `analytics`. 2. (Optional) To change the users able to manage this connection, change the users or groups listed under _Connection Admins_. :::warning If you don't specify any user or group, nobody can manage the connection - not even admins. ::: 3. (Optional) To prevent users from querying any Snowflake data, change _Allow SQL Query_ to **No**. 4. (Optional) To prevent users from previewing any Snowflake data, change _Allow Data Preview_ to **No**. 5. At the bottom of the screen, click **Next** to proceed. ### Agent extraction method Atlan supports using a Secure Agent for fetching metadata from Snowflake. To use a Secure Agent, follow these steps: 1. Select the **Agent** tab. 2. Configure the Snowflake data source by adding the secret keys for your secret store. For details on the required fields, refer to the [Direct extraction](#direct-extraction-method) section. 3. Complete the Secure Agent configuration by following the instructions in the [How to configure Secure Agent for workflow execution](https://docs.atlan.com/llms/platform/secure-agent/configure-secure-agent-for-workflow-execution/llms.txt) guide. 4. Click **Next** after completing the configuration. ## Configure crawler :::warning When [modifying](https://docs.atlan.com/llms/catalog/connector-capabilities/manage-connectivity/llms.txt) an existing Snowflake connection, switching to a different [extraction method](https://docs.atlan.com/llms/connectors/snowflake/permissions/llms.txt) deletes and recreates all assets in the existing connection. If you'd like to change the extraction method, [contact Atlan support](https://docs.atlan.com/support/submit-request) for assistance. ::: Before running the Snowflake crawler, you can further configure it. You must select the _Extraction method_ you configured when you [set up Snowflake](https://docs.atlan.com/llms/connectors/snowflake/set-up-snowflake/llms.txt): - For **Information Schema** [method](https://docs.atlan.com/llms/connectors/snowflake/permissions/llms.txt), keep the default selection. - Change to **Account Usage** [method](https://docs.atlan.com/llms/connectors/snowflake/permissions/llms.txt) and specify the following: - _Database Name_ of the copied Snowflake database - _Schema Name_ of the copied `ACCOUNT_USAGE` schema - **Incremental extraction** Your Snowflake account or environment must be upgraded to bundle version [2023_01](https://docs.snowflake.com/en/release-notes/bcr-bundles/2023_01/bcr-891) or later, as earlier versions don't support this feature. Toggle incremental extraction to enable faster and more efficient metadata extraction. You can override the defaults for any of the remaining options: - For _Asset selection_, select a filtering option: - To select the assets you want to include in crawling, click **Include by hierarchy** and filter for assets down to the database or schema level. (The crawler defaults to all assets when none are specified.) - To have the crawler include _Databases_, _Schemas_, or _Tables & Views_ based on a naming convention, click **Include by regex** and specify a regular expression - for example, specifying `ATLAN_EXAMPLE_DB.*` for _Databases_ includes all the matching databases and their child assets. - To select the assets you want to exclude from crawling, click **Exclude by hierarchy** and filter for assets down to the database or schema level. (The crawler defaults to no exclusions when none are specified.)  - To have the crawler ignore _Databases_, _Schemas_, or _Tables & Views_ based on a naming convention, click **Exclude by regex** and specify a regular expression - for example, specifying `ATLAN_EXAMPLE_TABLES.*` for _Tables & Views_ excludes all the matching tables and views. :::warning Pipe characters (`|`) in regex exclude patterns aren't supported and cause a `PatternSyntaxException`, which fails the crawl workflow. For example, using `{"^(USER|ZZZ).*":[]}` triggers this error. As a work-around, please use **Exclude by hierarchy** mode instead to filter out specific assets. ::: - Click **+** to add more filters. If you add multiple filters, the crawler processes assets that match _all_ the filtering conditions you have set. - To exclude lineage for views in Snowflake, change _View Definition Lineage_ to **No**. - To [import tags from Snowflake to Atlan](https://docs.atlan.com/llms/connectors/snowflake/manage-snowflake-tags/llms.txt), change _Import Tags_ to **Yes**. Note the following: - If using the _Account Usage_ extraction method, [grant the same permissions](https://docs.atlan.com/llms/connectors/snowflake/permissions/llms.txt) as required for crawling Snowflake assets to import tags and push updated tags to Snowflake. - If using the _Information Schema_ extraction method, note that Snowflake [stores all tag objects](https://docs.snowflake.com/en/user-guide/object-tagging#discover-tags) in the `ACCOUNT_USAGE` schema. [Grant permissions on the account usage schema to import tags](https://docs.atlan.com/llms/connectors/snowflake/permissions/llms.txt) from Snowflake. :::warning Object tagging in Snowflake currently requires [Enterprise Edition or higher](https://docs.snowflake.com/en/user-guide/intro-editions#feature-edition-matrix). If your organization doesn't have Enterprise Edition or higher and you try to import Snowflake tags to Atlan, the Snowflake connection fails with an error - unable to retrieve tags. ::: - To replace literal values in the SQL text Atlan extracts with placeholders, change _Redact SQL Text_ to **True**. This covers view and function definitions, stored procedure bodies, semantic expressions, and pipe `COPY` statements. Atlan preserves query structure, table references, and column references, so lineage is unaffected. This option is off by default, applies to future runs only, and increases crawler runtime. :::warning On the crawler, _Redact SQL Text_ works only on direct connections. It has no effect if your connection uses the [Agent extraction method](#agent-extraction-method). For redaction behavior and limitations, see [Query redaction for Snowflake](https://docs.atlan.com/llms/connectors/snowflake/query-redaction/llms.txt). ::: - For _Control Config_, keep _Default_ for the default configuration or click **Custom** to further configure the crawler: - If you have received a custom crawler configuration from Atlan support, for _Custom Config_, enter the value provided. You can also: - Enter `{"ignore-all-case": true}` to enable crawling assets with case-sensitive identifiers. - For _Enable Source Level Filtering_, click **True** to enable schema-level filtering at source or keep _False_ to disable it. - For _Use JDBC Internal Methods_, click **True** to enable JDBC internal methods for data extraction or click **False** to disable it. - For _Exclude tables with empty data_, change to **Yes** to exclude any tables and corresponding columns without any data. - For _Exclude views_, change to **Yes** to exclude all views from crawling. - To stop syncing Snowflake comments into asset descriptions, change _Sync Comments to Descriptions_ to **No**. When enabled (default), Snowflake `COMMENT` fields for tables and views are synced into the `description` field in Atlan on each crawl. This doesn't affect `userDescription` (set directly in Atlan), which takes display precedence when populated. Disable this option if you prefer to manage descriptions directly in Atlan. :::info **Did you know?** If an asset appears in both the include and exclude filters, the exclude filter takes precedence. ::: ## Run crawler To run the Snowflake crawler, after completing the preceding steps: 1. To check for any [permissions or other configuration issues](https://docs.atlan.com/llms/connectors/snowflake/preflight-checks-for-snowflake/llms.txt) before running the crawler, click **Preflight checks**. 2. You can either: - To run the crawler once immediately, at the bottom of the screen, click the **Run** button. - To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule Run** button. Once the crawler has completed running, the assets appear in Atlan's asset page! 🎉 Note that the Atlan crawler currently skips any unsupported data types so workflows complete successfully. ---