What does Atlan crawl from Dremio
Atlan crawls comprehensive metadata from your Dremio data lakehouse, including physical datasets, virtual datasets, folders, spaces, and sources information.
Assets
Atlan crawls the following Dremio assets, each with specific metadata fields.
DremioSpace
Dremio Spaces represent logical workspaces where users create and organize virtual datasets.
| Source field | Atlan field | Description |
|---|---|---|
name | name | Space name |
description | description | Space description |
tags | dremioLabels | Labels/tags associated with the space |
id | dremioId | Unique identifier for the space in Dremio |
DremioSource
Dremio Sources represent external data connections providing access to various systems.
| Source field | Atlan field | Description |
|---|---|---|
name | name | Source name |
description | description | Source description |
tags | dremioLabels | Labels/tags associated with the source |
type | dremioSourceType | Type of external source connection |
config | dremioSourceConnectionConfigs | Configuration parameters for connecting to the external source |
accelerationGracePeriodMs, accelerationRefreshPeriodMs, accelerationRefreshSchedule, accelerationActivePolicyType, accelerationNeverExpire, accelerationNeverRefresh | dremioSourceAccelerationSettings | Acceleration settings including grace period, refresh settings, and policy configuration |
metadataPolicy | dremioSourceMetadataPolicies | Metadata policy settings for the source |
id | dremioId | Unique identifier for the source in Dremio |
DremioFolder
Dremio Folders represent organizational containers within spaces or sources.
| Source field | Atlan field | Description |
|---|---|---|
name | name | Folder name |
description | description | Folder description |
tags | dremioLabels | Labels/tags associated with the folder |
id | dremioId | Unique identifier for the folder in Dremio |
Physical Dataset
Physical datasets represent tables from connected data sources available in Dremio.
| Source field | Atlan field | Description |
|---|---|---|
name | name | Physical dataset name |
description | description | Physical dataset description |
tags | dremioLabels | Labels/tags associated with the dataset |
rowCount | rowCount | Number of rows in the dataset |
columnCount | columnCount | Number of columns in the dataset |
createdAt | sourceCreatedAt | When the physical dataset was created |
id | dremioId | Unique identifier for the dataset in Dremio |
owner | sourceOwners | Owner of the dataset |
Virtual Dataset
Virtual datasets represent virtual tables created in Dremio through data virtualization.
| Source field | Atlan field | Description |
|---|---|---|
name | name | Virtual dataset name |
sql | definition | SQL definition of the virtual dataset |
description | description | Virtual dataset description |
tags | dremioLabels | Labels/tags associated with the dataset |
createdAt | sourceCreatedAt | When the virtual dataset was created |
id | dremioId | Unique identifier for the dataset in Dremio |
rowCount | rowCount | Number of rows in the dataset |
columnCount | columnCount | Number of columns in the dataset |
owner | sourceOwners | Owner of the dataset |
Column
Columns represent individual fields within physical and virtual datasets.
| Source field | Atlan field | Description |
|---|---|---|
name | name | Column name |
type | dataType | Column data type |
isNullable | isNullable | Whether the column permits null values |
description | description | Column description |
tags | dremioLabels | Labels/tags associated with the column |
id | dremioId | Unique identifier for the column in Dremio |
isPrimaryKey | isPrimaryKey | Whether the column is a primary key |
isPartitionColumn | isPartitionColumn | Whether the column is a partition column |
ordinalPosition | ordinalPosition | Position of the column in the table |
precision | precision | Precision of the column |
scale | scale | Scale of the column |
owner | sourceOwners | Owner of the column |
createdAt | sourceCreatedAt | When the column was created |