Crawl BigID

||Connect docs via MCP

Configure the Atlan BigID workflow to crawl metadata from your BigID instance and discover privacy-related data assets in Atlan. This guide walks through setting up the workflow, configuring the connection, mapping data sources, and running the crawler.

Prerequisites

Before you begin, make sure you have:

• Set up a BigID system user account. If not, follow the Set up BigID guide for detailed instructions.
• Your BigID domain name and API token to configure the workflow. You generate/copy an API token from your BigID User Settings → API Tokens.
• The required permissions to configure and run the workflow:
  • Atlan: Admin or Workflow Admin permissions
  • BigID: System user with API access
• Reviewed the order of operations for workflow execution.
• If using the Agent extraction method: deploy and configure Self-Deployed Runtime in your environment. For setup, see Configure workflow execution.

Create crawler workflow

Create a new BigID crawler workflow in Atlan by selecting the BigID connector package, configuring your extraction method and connection details, and running the crawler to extract metadata.

1. In your Atlan workspace, click Connectors in the left sidebar.

   • If you are using the Old UI (Classic), click New Workflow in the top navigation.

2. Click Marketplace.

3. Search for BigID and select it.

4. Click Install.

5. Once installation completes, click Setup Workflow on the same tile.

Select extraction method

Choose how you want to extract metadata from BigID:

• Direct: Atlan connects directly to your BigID instance from Atlan Cloud.
• Agent: Self-Deployed Runtime executes metadata extraction within your organization's environment.

Select your preferred extraction method using the Extraction method radio button.

Configure credentials

Configure authentication based on your selected extraction method.

Direct

In Direct extraction, configure BigID credentials stored in Atlan.

1. In the Credential section, select or create a BigID credential:

   • Host FQDN: Enter the BigID domain name (for example, account.mybigid.com). For private-network setups, use the private DNS associated with the link.
   • Authentication: Select Personal Access Token. This is the only supported authentication method.
   • Personal Access Token: Enter the API token created for the system user in the Set up BigID guide.
   • SSL certificate: If your BigID instance uses a self-signed SSL certificate, enter the root certificate PEM value.

2. Click Test Authentication to confirm connectivity to BigID.

3. When the test is successful, at the bottom of the screen click Next.

Agent

In Agent extraction, Self-Deployed Runtime executes metadata extraction within your organization's environment.

1. Install Self-Deployed Runtime if you haven't already:

   • Install via Docker Compose
   • Install on Kubernetes

2. Select the Agent tab.

3. Store sensitive information in the secret store configured with the Self-Deployed Runtime and reference the secrets in the corresponding fields. For more information, see Secret management.

4. For details on individual fields, refer to the Direct extraction tab.

5. Click Next after completing the configuration.

Configure connection

Set up the connection details and specify who can manage this connection.

1. Provide a Connection name that represents your source environment. For example, you might use values like production, development, gold, or analytics.

2. To change the users able to manage this connection, update the users or groups listed under Connection Admins.

3. At the bottom of the screen, click Next to proceed.

Configure connection mappings and metadata

On the Mapping & Configuration page, map BigID datasources to Atlan connections. The way you specify datasources differs based on your extraction method.

Direct

In Direct extraction, you can browse and select BigID datasources from a dropdown list.

1. In the Connection Mappings section, add a mapping row.

2. For each mapping:

   • Atlan Connection: Choose where the mapped BigID datasource's metadata lands. Open the dropdown and either:
     • Select an existing connection—a source Atlan already crawls (for example, your Snowflake or S3 connection). BigID metadata enriches those existing assets in place with classifications, custom metadata, and a data risk summary.
     • Create a new connection—click + Create new connection at the bottom of the dropdown. Atlan ingests the mapped BigID datasource's assets into this new connection. See What does Atlan crawl from BigID for how structured and unstructured sources are handled.
   • BigID Datasources: Click the dropdown to browse and select one or more BigID datasources for this mapping. The dropdown is populated by querying the BigID metadata endpoint. If the list appears empty, click the refresh icon.

3. To add another mapping, click Add to Connection Mappings and repeat the previous step.

Unstructured sources are ingested to the data lakehouse

When you ingest an unstructured BigID datasource into a new connection, Atlan catalogs its container and folder structure for exploration, but the individual objects (files) are written to Atlan's data lakehouse rather than the browsable catalog. File-level BigID context is retained for downstream governance, but the objects themselves don't appear as browsable assets in the Atlan UI. Structured sources are ingested in full and are browsable in Atlan.

Agent

In Agent extraction, you must enter exact BigID datasource names manually as comma-separated values since the metadata endpoint isn't available.

1. In the Connection Mappings section, add a mapping row.

2. For each mapping:

   • Atlan Connection: Select an existing connection to enrich its assets in place, or create a new connection to ingest the mapped datasource's assets. See the Direct extraction tab for details on the two behaviors.
   • BigID Datasources: Enter the exact BigID datasource names as comma-separated values in the text field.
     • For a single datasource: datasource_name
     • For multiple datasources: datasource_name_1, datasource_name_2, datasource_name_3

3. To add another mapping, click Add to Connection Mappings and repeat the previous step.

4. Run the crawler:
   • For Direct extraction: To verify permissions and configuration before running, click Preflight checks. After the checks pass, choose your run option.
   • For Agent extraction: Proceed directly to choosing your run option.
   • Choose your run option:
     • To run the crawler once immediately, click Run at the bottom of the screen.
     • To schedule the crawler to run hourly, daily, weekly, or monthly, click Schedule Run at the bottom of the screen.

Once the crawler completes, you can view the assets in Atlan's asset page.

Need help

• Contact Atlan support: For issues related to Atlan integration, contact Atlan support.

See also

• What does Atlan crawl from BigID: Learn what assets and metadata Atlan extracts from BigID.