Skip to main content

Crawl BigID

Configure the Atlan BigID workflow to crawl metadata from your BigID instance and discover privacy-related data assets in Atlan. This guide walks through setting up the workflow, configuring the connection, mapping data sources, and running the crawler.

Prerequisites​

Before you begin, you must have:

  • Set up a BigID system user account. If not, follow the Set up BigID guide for detailed instructions.

  • Your BigID domain name and API token to configure the workflow. You generate/copy an API token from your BigID User Settings → API Tokens.

  • The required permissions to configure and run the workflow:

    • Atlan: Admin or Workflow Admin permissions
    • BigID: System user with API access

  • An understanding of how the BigID workflow runs with other workflows by reviewing the order of operations.

Create crawler workflow​

Follow these steps in Atlan to create a workflow that crawls metadata from BigID:

  1. In Atlan, select New > New Workflow.
  2. From the package list, select BigID.
  3. Click Setup Workflow.

Configure credential​

Configure the BigID credential to authenticate with your BigID instance.

  1. In the Credential section, enter these details:

    • Host FQDN: Enter the BigID domain name (for example, account.mybigid.com). For private-network setups, use the private DNS associated with the link.

    • Authentication: Select Personal Access Token. This is the only supported authentication method.

    • Personal Access Token: Enter the API token created for the system user in the Set up BigID guide.

    • SSL certificate: If your BigID instance uses a self-signed SSL certificate, enter the root certificate PEM value.

    • Atlan API Key: Enter your Atlan API token. For information on generating an API token, see API tokens.

  2. Click Test Authentication to confirm connectivity to BigID.

  3. When the test is successful, at the bottom of the screen click Next.

Configure connection​

Set up the connection details and specify who can manage this connection.

  1. Provide a Connection name that represents your source environment. For example, you might use values like production, development, gold, or analytics.

  2. To change the users able to manage this connection, update the users or groups listed under Connection Admins.

  3. At the bottom of the screen, click Next to proceed.

Configure mappings​

Map BigID datasources to Atlan connections to establish the relationship between your data assets.

  1. In the Connection mappings section, click Add mapping to create a new mapping.

  2. For each mapping:

    • Atlan connection: Select the Atlan connection that houses the data assets that you want to enrich with BigID metadata.
    • BigID datasource: Select one or more BigID datasources that contain assets associated with the mapped Atlan connection.

  3. To add more mappings, click Add mapping again and repeat the previous step.

  4. When you finish adding mappings, click Next to configure custom metadata.

Configure custom metadata​

Set up custom metadata to store BigID-discovered attributes in Atlan.

  1. In Atlan, create a new custom metadata set named BigID Metadata with a text-based property named Attributes. This metadata houses the BigID-discovered, scan-related attributes that the workflow brings into Atlan.

  2. In the workflow configuration:

    • Set Attribute Custom Metadata to BigID Metadata.
    • Set Attribute Custom Metadata Property to Attributes.

Run crawler​

After you complete the previous sections, run the BigID crawler to discover and import metadata into Atlan.

  1. To run the crawler once immediately, click Run.

  2. To schedule the crawler to run on a recurring basis (hourly, daily, weekly, or monthly), click Schedule run and choose your schedule.

Need help​

See also​