What does Atlan crawl from Databricks?
This reference details the assets and properties that Atlan crawls from your Databricks workspace.
Lineage
Atlan supports complete lineage tracking for the following Databricks assets:
- Tables
- Views
- Materialized views
- Volumes
Assets
Atlan crawls and maps the following assets and properties from Databricks.
The following properties aren't crawled by the System tables extraction method:
- Table properties:
partitionList,partitionCount - Column properties:
maxLength,precision
Databases
Atlan maps databases from Databricks to its Database asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_CATALOG | name | Asset profile and overview sidebar |
SCHEMA_COUNT | schemaCount | API only |
Schemas
Atlan maps schemas from Databricks to its Schema asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_SCHEMA | name | Asset profile and overview sidebar |
TABLE_COUNT | tableCount | Asset preview and profile |
VIEW_COUNT | viewsCount | Asset preview and profile |
TABLE_CATALOG | databaseName | Asset preview and profile |
Tables
Atlan maps tables from Databricks to its Table asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | Asset profile and overview sidebar |
REMARKS, DESCRIPTION | description | Asset profile and overview sidebar |
COLUMN_COUNT | columnCount | Asset profile and overview sidebar |
LOCATION | externalLocation | Overview sidebar |
FORMAT | externalLocationFormat | Overview sidebar |
OWNER | Created (in Databricks) | Properties sidebar |
CREATEDAT | sourceCreatedAt | Properties sidebar |
UPDATED_BY | Last updated | Properties sidebar |
LASTMODIFIED | sourceUpdatedAt | Properties sidebar |
PARTITIONS | isPartitioned, partitionCount, partitionList | API only |
Views
Atlan maps views from Databricks to its View asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | Asset profile and overview sidebar |
REMARKS | description | Asset profile and overview sidebar |
COLUMN_COUNT | columnCount | Asset profile and overview sidebar |
CREATETAB_STMT | definition | Asset profile and overview sidebar |
OWNER | Created (in Databricks) | Properties sidebar |
CREATEDAT | sourceCreatedAt | Properties sidebar |
UPDATED_BY | Last updated | Properties sidebar |
LASTMODIFIED | sourceUpdatedAt | Properties sidebar |
Materialized views
Atlan maps materialized views from Databricks to its MaterialisedView asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | Asset profile and overview sidebar |
REMARKS | description | Asset profile and overview sidebar |
COLUMN_COUNT | columnCount | Asset profile and overview sidebar |
CREATETAB_STMT | definition | Asset profile and overview sidebar |
OWNER | Created (in Databricks) | Properties sidebar |
CREATEDAT | sourceCreatedAt | Properties sidebar |
UPDATED_BY | Last updated | Properties sidebar |
LASTMODIFIED | sourceUpdatedAt | Properties sidebar |
Volumes
Atlan maps volumes from Databricks to its Volume asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
VOLUME_NAME | name | Asset profile and overview sidebar |
COMMENT | description | Asset profile and overview sidebar |
VOLUME_TYPE | databricksVolumeType | Asset profile and overview sidebar |
VOLUME_OWNER | databricksVolumeOwner | Properties sidebar |
CREATED | sourceCreatedAt | Properties sidebar |
CREATED_BY | sourceCreatedBy | Properties sidebar |
LAST_ALTERED | sourceUpdatedAt | Properties sidebar |
LAST_ALTERED_BY | sourceUpdatedBy | Properties sidebar |
STORAGE_LOCATION | databricksVolumeExternalLocation | Overview sidebar |
VOLUME_CATALOG | databaseName | Asset preview and profile |
VOLUME_SCHEMA | schemaName | Asset preview and profile |
Columns
To help you work seamlessly with STRUCT data types, Atlan supports nested columns up to level 30 in Databricks. You can view these columns in the Tree view or the asset sidebar of your table assets, and also explore child columns of STRUCTs nested within MAPs or ARRAYs. However, lineage for nested columns isn't supported.
Atlan maps columns from Databricks to its Column asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
PRIMARY KEY | isPrimary | Asset preview and filter, overview sidebar |
FOREIGN KEY | isForeign | Asset preview and filter, overview sidebar |
COLUMN_NAME | name | Asset profile and overview sidebar |
REMARKS | description | Asset profile and overview sidebar |
ORDINAL_POSITION | order | Asset profile |
TYPE_NAME | dataType | Asset profile and overview sidebar |
PARTITION_INDEX | isPartition | Asset preview and profile |
NULLABLE | isNullable | API only |
CHAR_OCTET_LENGTH | maxLength | API only |
DECIMAL_DIGITS | precision | API only |