## Crawl BigID

URL: https://docs.atlan.com/apps/connectors/privacy/bigid/how-tos/crawl-bigid

> Configure and run the Atlan BigID workflow to crawl metadata from BigID.

Configure the Atlan BigID workflow to crawl metadata from your BigID instance and discover privacy-related data assets in Atlan. This guide walks through setting up the workflow, configuring the connection, mapping data sources, and running the crawler.

## Prerequisites

Before you begin, make sure you have:

- Set up a BigID system user account. If not, follow the [Set up BigID](https://docs.atlan.com/llms/connectors/bigid/set-up-bigid/llms.txt) guide for detailed instructions.
- Your BigID domain name and API token to configure the workflow. You generate/copy an API token from your BigID User Settings → API Tokens.
- The required permissions to configure and run the workflow:
 - **Atlan**: Admin or Workflow Admin permissions
 - **BigID**: System user with API access
- Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for workflow execution.
- If using the Agent extraction method: deploy and configure [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt) in your environment. For setup, see [Configure workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).

## Create crawler workflow

Create a new BigID crawler workflow in Atlan by selecting the BigID connector package, configuring your extraction method and connection details, and running the crawler to extract metadata.

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **BigID** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Select extraction method

Choose how you want to extract metadata from BigID:

- **Direct**: Atlan connects directly to your BigID instance from Atlan Cloud.
- **Agent**: Self-Deployed Runtime executes metadata extraction within your organization's environment.

Select your preferred extraction method using the **Extraction method** radio button.

### Configure credentials

Configure authentication based on your selected extraction method.

### Direct

In **Direct** extraction, configure BigID credentials stored in Atlan.

1. In the **Credential** section, select or create a BigID credential:
 - **Host FQDN**: Enter the BigID domain name (for example, `account.mybigid.com`). For private-network setups, use the private DNS associated with the link.
 - **Authentication**: Select **Personal Access Token**. This is the only supported authentication method.
 - **Personal Access Token**: Enter the API token created for the system user in the [Set up BigID](https://docs.atlan.com/llms/connectors/bigid/set-up-bigid/llms.txt) guide.
 - **SSL certificate**: If your BigID instance uses a self-signed SSL certificate, enter the root certificate PEM value.

2. Click **Test Authentication** to confirm connectivity to BigID.

3. When the test is successful, at the bottom of the screen click **Next**.

### Agent

In **Agent** extraction, Self-Deployed Runtime executes metadata extraction within your organization's environment.

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
2. Select the **Agent** tab.

3. Store sensitive information in the secret store configured with the Self-Deployed Runtime and reference the secrets in the corresponding fields. For more information, see [Secret management](https://docs.atlan.com/llms/platform/self-deployed-runtime/secret-management/llms.txt).

4. For details on individual fields, refer to the Direct extraction tab.

5. Click **Next** after completing the configuration.

### Configure connection

Set up the connection details and specify who can manage this connection.

1. Provide a _Connection name_ that represents your source environment. For example, you might use values like `production`, `development`, `gold`, or `analytics`.

2. To change the users able to manage this connection, update the users or groups listed under _Connection Admins_.

3. At the bottom of the screen, click **Next** to proceed.

### Configure connection mappings and metadata

On the **Mapping & Configuration** page, map BigID datasources to Atlan connections. The way you specify datasources differs based on your extraction method.

### Direct

In **Direct** extraction, you can browse and select BigID datasources from a dropdown list.

1. In the **Connection Mappings** section, add a mapping row.

2. For each mapping:
 - **Atlan Connection**: Choose where the mapped BigID datasource's metadata lands. Open the dropdown and either:
 - **Select an existing connection**—a source Atlan already crawls (for example, your Snowflake or S3 connection). BigID metadata *enriches* those existing assets in place with classifications, custom metadata, and a data risk summary.
 - **Create a new connection**—click **+ Create new connection** at the bottom of the dropdown. Atlan *ingests* the mapped BigID datasource's assets into this new connection. See [What does Atlan crawl from BigID](https://docs.atlan.com/llms/connectors/bigid/what-does-atlan-crawl-from-bigid/llms.txt) for how structured and unstructured sources are handled.
 - **BigID Datasources**: Click the dropdown to browse and select one or more BigID datasources for this mapping. The dropdown is populated by querying the BigID metadata endpoint. If the list appears empty, click the refresh icon.
3. To add another mapping, click **Add to Connection Mappings** and repeat the previous step.

:::note Unstructured sources are ingested to the data lakehouse
When you ingest an unstructured BigID datasource into a new connection, Atlan catalogs its container and folder structure for exploration, but the individual objects (files) are written to Atlan's data lakehouse rather than the browsable catalog. File-level BigID context is retained for downstream governance, but the objects themselves don't appear as browsable assets in the Atlan UI. Structured sources are ingested in full and are browsable in Atlan.
:::

### Agent

In **Agent** extraction, you must enter exact BigID datasource names manually as comma-separated values since the metadata endpoint isn't available.

1. In the **Connection Mappings** section, add a mapping row.

2. For each mapping:
 - **Atlan Connection**: Select an existing connection to enrich its assets in place, or create a new connection to ingest the mapped datasource's assets. See the Direct extraction tab for details on the two behaviors.
 - **BigID Datasources**: Enter the exact BigID datasource names as comma-separated values in the text field.
 - For a single datasource: `datasource_name`
 - For multiple datasources: `datasource_name_1, datasource_name_2, datasource_name_3`
3. To add another mapping, click **Add to Connection Mappings** and repeat the previous step.

4. Run the crawler:
 - **For Direct extraction**: To verify permissions and configuration before running, click **Preflight checks**. After the checks pass, choose your run option.
 - **For Agent extraction**: Proceed directly to choosing your run option.
 - Choose your run option:
 - To run the crawler once immediately, click **Run** at the bottom of the screen.
 - To schedule the crawler to run hourly, daily, weekly, or monthly, click **Schedule Run** at the bottom of the screen.

Once the crawler completes, you can view the assets in Atlan's asset page.

## Need help

- **Contact Atlan support**: For issues related to Atlan integration, [contact Atlan support](https://docs.atlan.com/support/submit-request).

## See also

- [What does Atlan crawl from BigID](https://docs.atlan.com/llms/connectors/bigid/what-does-atlan-crawl-from-bigid/llms.txt): Learn what assets and metadata Atlan extracts from BigID.

---