What does Atlan crawl from Dremio

✦TL;DR

Learn what assets and metadata Atlan crawls from Dremio data lakehouse.

||

|Connect

Atlan crawls metadata from your Dremio data lakehouse—physical datasets, virtual datasets, folders, spaces, and sources—and builds column-level lineage from each virtual dataset's SQL definition.

Assets

Atlan crawls the following Dremio assets, each with specific metadata fields.

DremioSpace

Dremio Spaces represent logical workspaces where users create and organize virtual datasets.

Source field	Atlan field	Description
name	name	Space name
id	dremioId	Unique identifier for the space in Dremio

DremioSource

Dremio Sources represent external data connections providing access to various systems.

Source field	Atlan field	Description
name	name	Source name
type	dremioSourceType	Type of external source connection
config	dremioSourceConnectionConfigs	Configuration parameters for connecting to the external source
accelerationGracePeriodMs, accelerationRefreshPeriodMs, accelerationRefreshSchedule, accelerationActivePolicyType, accelerationNeverExpire, accelerationNeverRefresh	dremioSourceAccelerationSettings	Acceleration settings including grace period, refresh settings, and policy configuration
metadataPolicy	dremioSourceMetadataPolicies	Metadata policy settings for the source
id	dremioId	Unique identifier for the source in Dremio

DremioFolder

Dremio Folders represent organizational containers within spaces or sources.

Source field	Atlan field	Description
name	name	Folder name
id	dremioId	Unique identifier for the folder in Dremio

Physical Dataset

Physical datasets represent tables from connected data sources available in Dremio.

Source field	Atlan field	Description
name	name	Physical dataset name
rowCount	rowCount	Number of rows in the dataset
columnCount	columnCount	Number of columns in the dataset
createdAt	sourceCreatedAt	When the physical dataset was created
id	dremioId	Unique identifier for the dataset in Dremio

Virtual Dataset

Virtual datasets represent virtual tables created in Dremio through data virtualization.

Source field	Atlan field	Description
name	name	Virtual dataset name
sql	definition	SQL definition of the virtual dataset
createdAt	sourceCreatedAt	When the virtual dataset was created
id	dremioId	Unique identifier for the dataset in Dremio
rowCount	rowCount	Number of rows in the dataset
columnCount	columnCount	Number of columns in the dataset

Column

Columns represent individual fields within physical and virtual datasets.

Source field	Atlan field	Description
name	name	Column name
type	dataType	Column data type
isNullable	isNullable	Whether the column permits null values
id	dremioId	Unique identifier for the column in Dremio
isPrimaryKey	isPrimaryKey	Whether the column is a primary key
isPartitionColumn	isPartitionColumn	Whether the column is a partition column
ordinalPosition	ordinalPosition	Position of the column in the table
precision	precision	Precision of the column
scale	scale	Scale of the column

Lineage

Atlan builds lineage for Dremio by parsing each virtual dataset's SQL definition:

• Virtual-to-physical (intra-Dremio): each virtual dataset is linked to the physical datasets—and upstream virtual datasets—it selects from, including multi-hop virtual-to-virtual chains.
• Cross-connector upstream: a Dremio physical dataset is linked to the real source table already in Atlan—AWS Glue and Iceberg tables (lakehouse) and relational tables such as Snowflake, PostgreSQL, or Redshift—when you select the source connection under Advanced Configuration during crawler setup and that source is crawled in Atlan.

Atlan also builds column-level lineage for Dremio:

• Source column to physical dataset column: a physical dataset mirrors its source table, so each of its columns is linked to the matching column on the upstream source asset in Atlan.
• Column to virtual dataset column: each virtual dataset's SQL is parsed per column, so a virtual dataset column is linked to every column it's derived from, including through expressions and joins.

Requirements:

• The crawling role needs SELECT on the datasets whose SQL is parsed (granted during setup—reading each view's SQL definition from INFORMATION_SCHEMA.VIEWS).
• For cross-connector upstream lineage, table-level and column-level, the upstream source connection must already exist in Atlan, and you must select it under Advanced Configuration.
• The virtual dataset and the datasets it selects from must be included in the same crawl. Assets outside the crawl's scope are skipped.
• Column-level lineage requires the virtual dataset's SQL to name its columns. A virtual dataset defined as SELECT * receives table-level lineage only.