Skip to main content

Crawl BigID

Configure the Atlan BigID workflow to crawl metadata from your BigID instance and discover privacy-related data assets in Atlan. This guide walks through setting up the workflow, configure connection, map data sources, and running the crawler.

Prerequisites​

Before you begin, make sure you have:

  • Set up a BigID system user account. If not, follow the Set up BigID guide for detailed instructions.
  • Your BigID domain name and API token which is needed to configure the workflow.
  • To crawl metadata from BigID, review the order of operations.

Permissions required​

To successfully configure the BigID workflow, make sure that your user role has below permissions:

  • Atlan: Admin or Workflow Admin permissions
  • BigID: System user with API access

Set up workflow​

Follow these steps in Atlan to create a BigID workflow:

  • Click New and then click New Workflow to set up a new workflow
  • From the list of packages, select BigID and click on Setup Workflow. Provide below details:
    • Workflow Name: Enter a unique name to help recognize and manage the workflow
    • Host FQDN: Enter the BigID domain name. For private-network setups, use the private DNS associated with the link.
    • Personal Access Token Value: Provide the API token created for the system user in the Set up BigID
    • SSL certificate: Enter the root certificate PEM value if your BigID instance exposes a self-signed certificate
  • Click Test Authentication to confirm connectivity to BigID
  • When successful, at the bottom of the screen click Next

Set up connection​

Set up the connection details and specify who can manage this connection. Follow these steps to configure the connection:

  • Provide a Connection Name that represents your source environment. For example, you might use values like production, development, gold, or analytics
  • (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins. If you don't specify any user or group, nobody can manage the connection - not even admins.
  • At the bottom of the screen, click Next to proceed

Map data sources​

Map BigID Data Sources to Atlan Connections to establish the relationship between your data assets. Follow these steps to map data sources:

  • Atlan Connection: Select the Atlan Connection that houses the data assets that you're trying to bring BigID metadata for
  • BigID Datasources: Select one or more BigID Data Sources that contain assets associated with the mapped Atlan Connection
  • Click Next to configure custom metadata

Configure custom metadata​

Set up custom metadata to store BigID-discovered attributes in Atlan. Follow these steps to configure custom metadata:

  • Custom Metadata: Create a new Custom Metadata on Atlan named BigID Metadata with a text-based property named Attributes. This is used to house the BigID-discovered, scan-related attributes that the workflow brings over to Atlan
  • Attribute Custom Metadata: Enter the value as BigID Metadata
  • Attribute Custom Metadata Property: Enter the value as Attributes

Run crawler​

Execute the BigID crawler to discover and import metadata. Follow these steps to run crawler:

  • Run immediately: Click the Run button to run the crawler once immediately
  • Schedule run: Click the Schedule Run button to schedule the crawler to run hourly, daily, weekly, or monthly

Troubleshooting​

If you encounter issues during the BigID crawl process:

  • Authentication issues: Verify your API token is valid and has the correct permissions
  • SSL certificate errors: Check that you've provided the correct root certificate PEM value
  • No metadata appears: Check that data source mapping is correct and BigID contains assets for the mapped connections

Need help​

See also​