What does Atlan crawl from BigID?
Once you have crawled BigID, you can use connector-specific filters for quick asset discovery. This document provides details on the metadata and assets that Atlan crawls from BigID.
Enrichment and ingestion
Atlan brings BigID metadata into your workspace in one of two ways, depending on the Atlan connection you map each BigID datasource to during crawl configuration:
- Enrichment—when you map a BigID datasource to an existing Atlan connection (a source Atlan already crawls), Atlan matches the corresponding assets and applies BigID-discovered metadata to them in place. Assets that don't already exist in the mapped connection are skipped.
- Ingestion—when you map a BigID datasource to a new connection, Atlan ingests the datasource's assets into that connection so they appear in your catalog.
In both cases, the BigID metadata Atlan applies is the same: attributes, tags, a data risk summary, and policy violations.
Ingested assets
When you ingest a BigID datasource into a new connection, how its assets appear in Atlan depends on whether the source is structured or unstructured.
- Structured sources (for example, databases) are ingested in full. Atlan catalogs the database, schema, table, and column assets, and they're browsable and searchable in the Atlan UI.
- Unstructured sources are ingested to the container and folder level. Atlan catalogs the container and folder structure, but the individual objects (files) within them are written to Atlan's data lakehouse rather than the browsable catalog.
File-level BigID context for unstructured objects is retained in Atlan's data lakehouse for downstream governance, but the objects themselves don't appear as browsable assets in the Atlan UI.
Attributes
Atlan maps BigID Attributes, associated with Catalog Objects as a result of scans, to a standard Custom Metadata entity named BigID Attributes. Multiple attributes are concatenated with comma as the delimiter.
Tags
Atlan sources both system-assigned and user-assigned tags on BigID Catalog Objects to source-linked Atlan Tags on the associated Atlan assets. The sourced tags can then be propagated and assigned like regular Atlan Tags.
Assigning values for BigID-sourced Tags on Atlan and reverse-sync of those values back to BigID is currently not supported.
Data risk summary
For each asset, Atlan ingests a data risk summary from BigID and surfaces it as the asset's summary in Atlan. The summary is a rollup of BigID's findings for that asset—for example, classification and attribute counts and policy-violation findings—and is searchable from the Assets page. Where BigID exposes it, the summary also links to the asset's page in the BigID catalog.
Policy violations
Atlan maps Policy violations detected for Catalog Objects during BigID scans as Announcements on Atlan. The Announcement messages indicate the details of the Policies found violated. Once these violations are marked resolved on BigID, the corresponding Atlan Announcements are updated or removed appropriately.