Crawl BigID

Configure the Atlan BigID workflow to crawl metadata from your BigID instance and discover privacy-related data assets in Atlan. This guide walks through setting up the workflow, configuring the connection, mapping data sources, and running the crawler.

Prerequisites

Before you begin, make sure you have:

• Set up a BigID system user account. If not, follow the Set up BigID guide for detailed instructions.
• Your BigID domain name and API token to configure the workflow. You generate/copy an API token from your BigID User Settings → API Tokens.
• The required permissions to configure and run the workflow:
  • Atlan: Admin or Workflow Admin permissions
  • BigID: System user with API access
• Reviewed the order of operations for workflow execution.
• If using Agent extraction method: Self-Deployed Runtime deployed and configured. Complete the Secure Agent configuration by following the instructions in the How to configure Secure Agent for workflow execution guide.

Create crawler workflow

Create a new BigID crawler workflow in Atlan by selecting the BigID connector package, configuring your extraction method and connection details, and running the crawler to extract metadata.

1. In the top right of any screen, navigate to New and then click New Workflow.
2. From the list of packages, select BigID and click Setup Workflow.

Select extraction method

Choose how you want to extract metadata from BigID:

• Direct: Atlan connects directly to your BigID instance from Atlan Cloud.
• Agent: Self-Deployed Runtime executes metadata extraction within your organization's environment.

Select your preferred extraction method using the Extraction method radio button.

Configure credentials

Configure authentication based on your selected extraction method.

Direct

In Direct extraction, configure BigID credentials stored in Atlan.

1. In the Credential section, select or create a BigID credential:

   • Host FQDN: Enter the BigID domain name (for example, account.mybigid.com). For private-network setups, use the private DNS associated with the link.
   • Authentication: Select Personal Access Token. This is the only supported authentication method.
   • Personal Access Token: Enter the API token created for the system user in the Set up BigID guide.
   • SSL certificate: If your BigID instance uses a self-signed SSL certificate, enter the root certificate PEM value.
   • Atlan API Key: Enter your Atlan API token. For information on generating an API token, see API tokens.

2. Click Test Authentication to confirm connectivity to BigID.

3. When the test is successful, at the bottom of the screen click Next.

Agent

In Agent extraction, Self-Deployed Runtime executes metadata extraction within your organization's environment.

1. Install Self-Deployed Runtime if you haven't already.
2. Select the Agent tab.
3. Store sensitive information in the secret store configured with the Self-Deployed Runtime and reference the secrets in the corresponding fields. For more information, see Configure secrets for workflow execution.
4. For details on individual fields, refer to the Direct extraction tab.
5. Click Next after completing the configuration.

Configure connection

Set up the connection details and specify who can manage this connection.

1. Provide a Connection name that represents your source environment. For example, you might use values like production, development, gold, or analytics.

2. To change the users able to manage this connection, update the users or groups listed under Connection Admins.

3. At the bottom of the screen, click Next to proceed.

Configure connection mappings

Map BigID datasources to Atlan connections to establish the relationship between your data assets. The way you specify datasources differs based on your extraction method.

Direct

In Direct extraction, you can browse and select BigID datasources from a dropdown list.

1. In the Connection mappings section, click Add mapping to create a new mapping.
2. For each mapping:
   • Atlan connection: Select the Atlan connection that houses the data assets that you want to enrich with BigID metadata.
   • BigID datasource: Click the dropdown to browse and select one or more BigID datasources that contain assets associated with the mapped Atlan connection. The dropdown is populated by querying the BigID metadata endpoint.
3. To add more mappings, click Add mapping again and repeat the previous step.
4. When you finish adding mappings, click Next to configure custom metadata.

Agent

In Agent extraction, you must enter exact BigID datasource names manually as comma-separated values since the metadata endpoint isn't available.

1. In the Connection mappings section, click Add mapping to create a new mapping.
2. For each mapping:
   • Atlan connection: Select the Atlan connection that houses the data assets that you want to enrich with BigID metadata.
   • BigID datasource: Enter the exact BigID datasource names as comma-separated values in the text field.
     • For a single datasource: datasource_name
     • For multiple datasources: datasource_name_1, datasource_name_2, datasource_name_3
3. To add more mappings, click Add mapping again and repeat the previous step.
4. When you finish adding mappings, click Next to configure custom metadata.

Configure custom metadata

Set up custom metadata to store BigID-discovered attributes in Atlan.

1. In Atlan, create a new custom metadata set named BigID Metadata with a text-based property named Attributes. This metadata houses the BigID-discovered, scan-related attributes that the workflow brings into Atlan.

2. In the workflow configuration:

   • Set Attribute Custom Metadata to BigID Metadata.
   • Set Attribute Custom Metadata Property to Attributes.

Run crawler

Run preflight checks to validate your configuration, then execute the crawler immediately or schedule it to run on a recurring basis.

1. To verify permissions and configuration before running, click Preflight checks. This option is available for Direct extraction only.
2. Choose your run option:
   • To run the crawler once immediately, click Run at the bottom of the screen.
   • To schedule the crawler to run hourly, daily, weekly, or monthly, click Schedule Run at the bottom of the screen.

Once the crawler completes, you can view the assets in Atlan's asset page.

Need help

• Contact Atlan support: For issues related to Atlan integration, contact Atlan support.

See also

• What does Atlan crawl from BigID: Learn what assets and metadata Atlan extracts from BigID.