What does Atlan crawl from Google BigQuery?
Once you have crawled Google BigQuery, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:
- Tables - BigQuery labels and Is sharded filters
Atlan does not run any table scans. Atlan leverages the table preview options from Google BigQuery that allow you to view data for free and without affecting any quotas using the tabledata.list API. Hence, table asset previews in Atlan are already cost-optimized. However, this does not apply to views and materialized views.
For Google BigQuery views and materialized views, Atlan will send you a cost nudge before viewing a sample data preview. This will inform you about the precise bytes that will be spent during the execution of the query, helping you decide if you would still like to run the preview.
You will also receive a cost nudge before querying your Google BigQuery assets.
Atlan crawls and maps the following assets and properties from Google BigQuery.
Databases
Atlan maps projects from Google BigQuery to its Database asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
Project ID | name | asset preview, profile, and filter, overview sidebar |
Schemas
Atlan maps datasets from Google BigQuery to its Schema asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_SCHEMA | name | asset preview and profile, overview sidebar |
TABLE_COUNT | tableCount | asset preview and profile |
VIEW_COUNT | viewsCount | asset preview and profile, overview sidebar |
TABLE_CATALOG | databaseName | asset preview |
REMARKS | description | asset preview and profile, overview sidebar |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
Tables
Table asset previews are already cost-optimized. Google BigQuery allows you to use the table preview options to view data for free and without affecting any quotas. Note that this is currently not supported for Google BigQuery views and materialized views in Atlan.
Atlan maps tables from Google BigQuery to its Table asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
ROW_COUNT | rowCount | asset preview, profile, and filter, overview sidebar |
SIZE_BYTES | sizeBytes | asset filter and overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
LABELS | assetTags | overview sidebar |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
Views
Atlan maps views from Google BigQuery to its View asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
DDL | definition | asset profile and overview sidebar |
Materialized views
Atlan maps materialized views from Google BigQuery to its MaterialisedView asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
ROW_COUNT | rowCount | asset preview, profile, and filter, overview sidebar |
SIZE_BYTES | sizeBytes | asset filter and overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
DDL | definition | asset profile and overview sidebar |
Columns
Atlan supports nested columns up to level 1 for Google BigQuery to help you enrich your semi-structured data types. You can view nested columns in the asset sidebar for your table assets.
Atlan maps columns from Google BigQuery to its Column asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
COLUMN_NAME | name | asset preview and profile, overview sidebar |
REMARKS, DESCRIPTION | description | asset preview and profile, overview sidebar |
ORDINAL_POSITION | order | asset profile |
TYPE_NAME | dataType | asset preview, profile, and filter, overview sidebar |
IS_NULLABLE | isNullable | API only |
IS_PARTITIONING_COLUMN | isPartition | asset preview, profile, and filter |
CLUSTERING_COLUMN_LIST | isClustered | asset preview, profile, and filter |
Stored procedures
Atlan maps stored procedures from Google BigQuery to its Procedure asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
PROCEDURE_NAME | name | API only |
REMARKS | description | API only |
PROCEDURE_TYPE | subType | API only |
ROUTINE_DEFINITION | definition | API only |
CREATED | sourceCreatedAt | API only |
MODIFIED | sourceUpdatedAt | API only |