What does Atlan crawl from Google BigQuery?
Once you have crawled Google BigQuery, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:
- Tables - BigQuery labels and Is sharded filters
Atlan doesn't run any table scans. Atlan leverages the table preview options from Google BigQuery that enable you to view data for free and without affecting any quotas using the tabledata.list API. Hence, table asset previews in Atlan are already cost-optimized. However, this doesn't apply to views and materialized views.
For Google BigQuery views and materialized views, Atlan sends you a cost nudge before viewing a sample data preview. This informs you about the precise bytes that are spent during the execution of the query, helping you decide if you still want to run the preview.
You also receive a cost nudge before querying your Google BigQuery assets.
Atlan crawls and maps the following assets and properties from Google BigQuery.
Databases
Atlan maps projects from Google BigQuery to its Database asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
Project ID | name | asset preview, profile, and filter, overview sidebar |
Schemas
Atlan maps datasets from Google BigQuery to its Schema asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_SCHEMA | name | asset preview and profile, overview sidebar |
TABLE_COUNT | tableCount | asset preview and profile |
VIEW_COUNT | viewsCount | asset preview and profile, overview sidebar |
TABLE_CATALOG | databaseName | asset preview |
REMARKS | description | asset preview and profile, overview sidebar |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
Tables
Table asset previews are already cost-optimized. Google BigQuery enables you to use the table preview options to view data for free and without affecting any quotas. Note that this isn't currently supported for Google BigQuery views and materialized views in Atlan.
Atlan maps tables from Google BigQuery to its Table asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
ROW_COUNT | rowCount | asset preview, profile, and filter, overview sidebar |
SIZE_BYTES | sizeBytes | asset filter and overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
LABELS | assetTags | overview sidebar |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
Views
Atlan maps views from Google BigQuery to its View asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
DDL | definition | asset profile and overview sidebar |
Materialized views
Atlan maps materialized views from Google BigQuery to its MaterialisedView asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
ROW_COUNT | rowCount | asset preview, profile, and filter, overview sidebar |
SIZE_BYTES | sizeBytes | asset filter and overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
DDL | definition | asset profile and overview sidebar |
Columns
Atlan supports nested columns up to level 1 for Google BigQuery to help you enrich your semi-structured data types. You can view nested columns in the asset sidebar for your table assets. Atlan maps columns from Google BigQuery to its Column asset type.
Atlan doesn't crawl primary key (PK) and foreign key (FK) information from Google BigQuery.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
COLUMN_NAME | name | asset preview and profile, overview sidebar |
REMARKS, DESCRIPTION | description | asset preview and profile, overview sidebar |
ORDINAL_POSITION | order | asset profile |
TYPE_NAME | dataType | asset preview, profile, and filter, overview sidebar |
IS_NULLABLE | isNullable | API only |
IS_PARTITIONING_COLUMN | isPartition | asset preview, profile, and filter |
CLUSTERING_COLUMN_LIST | isClustered | asset preview, profile, and filter |
Routines
Atlan maps routines (user-defined functions, table-valued functions and stored procedures) from Google BigQuery to its BigqueryRoutine asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
ROUTINE_NAME | name | asset preview and profile, overview sidebar and filter |
REMARKS | description | asset preview and profile, overview sidebar |
ROUTINE_DEFINITION | definition | asset profile and overview sidebar |
CREATED | sourceCreatedAt | asset profile and overview sidebar |
MODIFIED | sourceUpdatedAt | asset profile and overview sidebar |
ROUTINE_TYPE | BigqueryRoutineType | asset preview and profile, overview sidebar and filter |
ROUTINE_ARGUMENTS | BigqueryRoutineArguments | asset profile and overview sidebar |