What does Atlan crawl from AWS Glue?
Atlan crawls and maps the following assets and properties from AWS Glue.
Databases
Atlan maps databases from AWS Glue to its Database asset type.
| Source property | Atlan property |
|---|---|
TABLE_CATALOG | name |
SCHEMA_COUNT | schemaCount |
Schemas
Atlan maps schemas from AWS Glue to its Schema asset type.
| Source property | Atlan property |
|---|---|
TABLE_SCHEMA | name |
TABLE_COUNT | tableCount |
VIEW_COUNT | viewsCount |
TABLE_CATALOG | databaseName |
COMMENTS | description |
CreateTime | sourceCreatedAt |
Tables
Atlan maps tables from AWS Glue to its Table asset type.
| Source property | Atlan property |
|---|---|
TABLE_NAME | name |
COMMENTS | description |
COLUMN_COUNT | columnCount |
ROW_COUNT | rowCount |
objectCount | tableObjectCount |
BYTES | sizeBytes |
Parameters (recordCount) | rowCount |
Parameters (sizeKey) | sizeBytes |
TABLE_TYPE | subType |
StorageDescriptor (Location) | externalLocation |
Parameters (typeOfData, classification) | externalLocationFormat |
PartitionKeys | isPartitioned |
PartitionData | partitionCount, partitionList |
CreatedBy | sourceCreatedBy |
CreateTime, CreationTime | sourceCreatedAt |
UpdateTime, LastAccessTime | sourceUpdatedAt |
Views
Atlan maps views from AWS Glue to its View asset type.
| Source property | Atlan property |
|---|---|
TABLE_NAME | name |
COMMENTS | description |
COLUMN_COUNT | columnCount |
VIEW_DEFINITION | definition |
Parameters (recordCount) | rowCount |
PartitionKeys | isPartitioned |
PartitionData | partitionCount, partitionList |
CreatedBy | sourceCreatedBy |
CreateTime, CreationTime | sourceCreatedAt |
UpdateTime, LastAccessTime | sourceUpdatedAt |
Columns
Atlan maps columns from AWS Glue to its Column asset type.
Atlan also supports nested columns up to level 15 for AWS Glue to help you enrich your semi-structured data types:
- Atlan retrieves raw
STRUCTandARRAYtype objects for nested columns up to 15 levels. - View nested columns in the column preview and overview sidebar for your table assets.
- Column-level lineage is supported. Search, enrich metadata, and view lineage for nested columns.
- Tag propagation is currently only supported from parent to nested columns.
- Atlan currently doesn't parse
MAPtype objects for columns and nested columns.
| Source property | Atlan property |
|---|---|
COLUMN_NAME | name |
COMMENTS | description |
ORDINAL_POSITION, COLUMN_ID | order |
TYPE_NAME, DATA_TYPE | dataType |
IS_PARTITION | isPartition |
PARTITION_ORDER | partitionOrder |