## What does Atlan crawl from Databricks?

URL: https://docs.atlan.com/apps/connectors/data-warehouses/databricks/references/what-does-atlan-crawl-from-databricks

> During a Databricks crawl, Atlan extracts metadata for Databases, Schemas, Tables, Views, Materialized Views, Volumes, External Locations, and AI Models. Reference tables map each Databricks property to its Atlan asset.

Atlan crawls Databricks metadata for databases, schemas, tables, views, materialized views, volumes, external locations, and AI models, mapping each to its Atlan asset. The tables below list which Databricks properties are read and how they appear on the corresponding Atlan asset.

## Lineage

Atlan supports complete lineage tracking for the following Databricks assets:

- **Tables**
- **Views**
- **Materialized views**
- **Metric views**
- **Volumes**
- **External locations**
- **AI model versions** (requires [additional permissions](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks-ai-models/llms.txt))
- **AI/BI dashboards** (private preview, requires [additional permissions](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks-ai-bi-dashboards/llms.txt))

## Assets

Atlan crawls and maps the following assets and properties from Databricks. 

:::info 
The following properties aren't crawled by the **System tables** extraction method:
 * **Table properties**: `partitionList`, `partitionCount`
 * **Column properties**: `maxLength`, `precision`
:::

### Databases

Atlan maps databases from Databricks to its `Database` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `TABLE_CATALOG` | `name` | Asset profile and overview sidebar |
| `SCHEMA_COUNT` | `schemaCount` | API only |

### Schemas

Atlan maps schemas from Databricks to its `Schema` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `TABLE_SCHEMA` | `name` | Asset profile and overview sidebar |
| `TABLE_COUNT` | `tableCount` | Asset preview and profile |
| `VIEW_COUNT` | `viewsCount` | Asset preview and profile |
| `TABLE_CATALOG` | `databaseName` | Asset preview and profile |

### Tables

Atlan maps tables from Databricks to its `Table` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `TABLE_NAME` | `name` | Asset profile and overview sidebar |
| `REMARKS, DESCRIPTION` | `description` | Asset profile and overview sidebar |
| `COLUMN_COUNT` | `columnCount` | Asset profile and overview sidebar |
| `LOCATION` | `externalLocation` | Overview sidebar |
| `FORMAT` | `externalLocationFormat` | Overview sidebar |
| `OWNER` | `Created (in Databricks)` | Properties sidebar |
| `CREATEDAT` | `sourceCreatedAt` | Properties sidebar |
| `UPDATED_BY` | `Last updated` | Properties sidebar |
| `LASTMODIFIED` | `sourceUpdatedAt` | Properties sidebar |
| `PARTITIONS` | `isPartitioned`, `partitionCount`, `partitionList` | API only |

### Views

Atlan maps views from Databricks to its `View` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `TABLE_NAME` | `name` | Asset profile and overview sidebar |
| `REMARKS` | `description` | Asset profile and overview sidebar |
| `COLUMN_COUNT` | `columnCount` | Asset profile and overview sidebar |
| `CREATETAB_STMT` | `definition` | Asset profile and overview sidebar |
| `OWNER` | `Created (in Databricks)` | Properties sidebar |
| `CREATEDAT` | `sourceCreatedAt` | Properties sidebar |
| `UPDATED_BY` | `Last updated` | Properties sidebar |
| `LASTMODIFIED` | `sourceUpdatedAt` | Properties sidebar |

### Metric views

Atlan maps metric views from Databricks to its `DatabricksMetricView` asset type.

:::warning **Note** 
 **upstream column lineage** for dimensions/measures **isn't** supported.
:::

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `TABLE_NAME` | `name` | Asset profile and overview sidebar |
| `REMARKS` | `description` | Asset profile and overview sidebar |
| `COLUMN_COUNT` | `columnCount` | Asset profile and overview sidebar |
| `CREATETAB_STMT` | `definition` | Asset profile and overview sidebar |
| `OWNER` | `Created (in Databricks)` | Properties sidebar |
| `CREATEDAT` | `sourceCreatedAt` | Properties sidebar |
| `UPDATED_BY` | `Last updated` | Properties sidebar |
| `LASTMODIFIED` | `sourceUpdatedAt` | Properties sidebar |

### Materialized views

Atlan maps materialized views from Databricks to its `MaterialisedView` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `TABLE_NAME` | `name` | Asset profile and overview sidebar |
| `REMARKS` | `description` | Asset profile and overview sidebar |
| `COLUMN_COUNT` | `columnCount` | Asset profile and overview sidebar |
| `CREATETAB_STMT` | `definition` | Asset profile and overview sidebar |
| `OWNER` | `Created (in Databricks)` | Properties sidebar |
| `CREATEDAT` | `sourceCreatedAt` | Properties sidebar |
| `UPDATED_BY` | `Last updated` | Properties sidebar |
| `LASTMODIFIED` | `sourceUpdatedAt` | Properties sidebar |

### External location

Atlan maps external locations from Databricks to its `DatabricksExternalLocation` asset type. 

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `EXTERNAL_LOCATION_NAME` | `name` | Asset profile and overview sidebar |
| `COMMENT` | `description` | Asset profile and overview sidebar |
| `EXTERNAL_LOCATION_OWNER` | `databricksOwner` | Properties sidebar |
| `CREATED` | `sourceCreatedAt` | Properties sidebar |
| `CREATED_BY` | `sourceCreatedBy` | Properties sidebar |
| `LAST_ALTERED` | `sourceUpdatedAt` | Properties sidebar |
| `LAST_ALTERED_BY` | `sourceUpdatedBy` | Properties sidebar |
| `URL` | `databricksUrl` | Overview sidebar |

### Volumes

Atlan maps volumes from Databricks to its `Volume` asset type. 

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `VOLUME_NAME` | `name` | Asset profile and overview sidebar |
| `COMMENT` | `description` | Asset profile and overview sidebar |
| `VOLUME_TYPE` | `databricksVolumeType` | Asset profile and overview sidebar |
| `VOLUME_OWNER` | `databricksVolumeOwner` | Properties sidebar |
| `CREATED` | `sourceCreatedAt` | Properties sidebar |
| `CREATED_BY` | `sourceCreatedBy` | Properties sidebar |
| `LAST_ALTERED` | `sourceUpdatedAt` | Properties sidebar |
| `LAST_ALTERED_BY` | `sourceUpdatedBy` | Properties sidebar |
| `STORAGE_LOCATION` | `databricksVolumeExternalLocation` | Overview sidebar |
| `VOLUME_CATALOG` | `databaseName` | Asset preview and profile |
| `VOLUME_SCHEMA` | `schemaName` | Asset preview and profile |

### Columns

:::info **Did you know?** 
 To help you work seamlessly with STRUCT data types, Atlan supports [nested columns up to level 30](https://docs.atlan.com/llms/connectors/databricks/troubleshooting-databricks-connectivity/llms.txt) in Databricks. You can view these columns in the Tree view or the asset sidebar of your table assets, and also explore child columns of STRUCTs nested within MAPs or ARRAYs. However, **lineage** for nested columns **isn't** supported.
:::

Atlan maps columns from Databricks to its `Column` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `PRIMARY KEY` | `isPrimary` | Asset preview and filter, overview sidebar |
| `FOREIGN KEY` | `isForeign` | Asset preview and filter, overview sidebar |
| `COLUMN_NAME` | `name` | Asset profile and overview sidebar |
| `REMARKS` | `description` | Asset profile and overview sidebar |
| `ORDINAL_POSITION` | `order` | Asset profile |
| `TYPE_NAME` | `dataType` | Asset profile and overview sidebar |
| `PARTITION_INDEX` | `isPartition` | Asset preview and profile |
| `NULLABLE` | `isNullable` | API only |
| `CHAR_OCTET_LENGTH` | `maxLength` | API only |
| `DECIMAL_DIGITS` | `precision` | API only |

## AI models

AI models are crawled only when [additional permissions are granted](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks-ai-models/llms.txt) and the Direct extraction strategy is used. Only models registered in the Unity Catalog Model Registry are crawled.

### AI models

Atlan maps AI models from Databricks to its `AIModel` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| Model name | `name` | Asset profile and overview sidebar |
| `COMMENT` | `description` | Asset profile and overview sidebar |
| Catalog | `databaseName` | Asset preview and profile |
| Schema | `schemaName` | Asset preview and profile |
| Owner | `sourceOwner` | Properties sidebar |
| Created at | `sourceCreatedAt` | Properties sidebar |

### AI model versions

Atlan maps AI model versions from Databricks to its `AIModelVersion` asset type.

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| Version | `modelVersionName` | Asset profile and overview sidebar |
| Version status | `modelVersionStatus` | Asset profile and overview sidebar |
| Source URI | `modelSourceURL` | Overview sidebar |
| Artifact URI | `modelArtifactPath` | Overview sidebar |
| Aliases | `modelVersionAliases` | Overview sidebar |
| Dataset count | `modelDatasetCount` | API only |
| Run ID | `modelRunId` | Properties sidebar |
| Run name | `modelRunName` | Properties sidebar |
| Run start time | `modelRunStartTime` | Properties sidebar |
| Run end time | `modelRunEndTime` | Properties sidebar |
| Metrics | `modelMetrics` | API only |
| Parameters | `modelParameters` | API only |

### AI/BI dashboards

Dashboard metadata comes from the Databricks Lakeview API and requires the **Import AI/BI Dashboards** option. See [Crawl Databricks AI/BI dashboards](https://docs.atlan.com/llms/connectors/databricks/crawl-databricks-ai-bi-dashboards/llms.txt).

| Source property | Atlan property | Where in Atlan |
|---|---|---|
| `display_name` | `name` | Asset profile and overview sidebar |
| `path` | `databricksDashboardPath` | Overview sidebar |
| `warehouse_id` | `databricksDashboardWarehouseId` | Overview sidebar |
| `lifecycle_state` | `databricksDashboardLifecycleState` | Properties sidebar |
| `etag` | `databricksDashboardEtag` | API only |
| `dashboard_id` | `sourceId` | Properties sidebar |
| Workspace identifier | `databricksDashboardWorkspaceId` | Properties sidebar |
| Genie space association | `databricksDashboardIsGenieSpaceEnabled` | Properties sidebar |
| Published dashboard link | `sourceURL` | Asset profile |
| `CAN MANAGE` user grants | `ownerUsers` | Overview sidebar |

---