What does Atlan crawl from Google BigQuery?
Atlan doesn't run any table scans. Atlan leverages the table preview options from [Google BigQuery](https://cloud.google.com/bigquery/docs/best-practices-costs#preview-data) that enable you to view data for free and without affecting any quotas using the `tabledata.list` API. Hence, [table](/apps/connectors/data-warehouses/google-bigquery/references/what-does-atlan-crawl-from-google-bigquery#tables) asset previews in Atlan are already cost-optimized. However, this doesn't apply to [views](/apps/connectors/data-warehouses/google-bigquery/references/what-does-atlan-crawl-from-google-bigquery#views) and [materialized views](/apps/connectors/data-warehouses/google-bigquery/references/what-does-atlan-crawl-from-google-bigquery#materialized-views).
Once you have crawled Google BigQuery, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:
- Tables - BigQuery labels and Is sharded filters
Atlan doesn't run any table scans. Atlan leverages the table preview options from Google BigQuery that enable you to view data for free and without affecting any quotas using the tabledata.list API. Hence, table asset previews in Atlan are already cost-optimized. However, this doesn't apply to views and materialized views.
For Google BigQuery views and materialized views, Atlan runs a query to generate the sample data preview. The preview query bills to the project determined by the Billing Strategy you configure in the miner: with the default Central Project strategy, preview queries bill to the Billing Project Id in your crawler credentials; with Each Mined Project, each view's preview query bills to that view's own project. Atlan shows you a cost nudge before running the preview so you can review the bytes to be processed before you commit to the query.
If you haven't configured a BigQuery miner, preview queries bill to the Billing Project Id configured in your crawler credentials.
You also receive a cost nudge before querying your Google BigQuery assets.
Atlan crawls and maps the following assets and properties from Google BigQuery.
Databases
Atlan maps projects from Google BigQuery to its Database asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
Project ID | name | asset preview, profile, and filter, overview sidebar |
Schemas
Atlan maps datasets from Google BigQuery to its Schema asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_SCHEMA | name | asset preview and profile, overview sidebar |
TABLE_COUNT | tableCount | asset preview and profile |
VIEW_COUNT | viewsCount | asset preview and profile, overview sidebar |
TABLE_CATALOG | databaseName | asset preview |
REMARKS | description | asset preview and profile, overview sidebar |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
Tables
Table asset previews are already cost-optimized. Google BigQuery enables you to use the table preview options to view data for free and without affecting any quotas. Note that this isn't currently supported for Google BigQuery views and materialized views in Atlan.
Atlan maps tables from Google BigQuery to its Table asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
ROW_COUNT | rowCount | asset preview, profile, and filter, overview sidebar |
SIZE_BYTES | sizeBytes | asset filter and overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
LABELS | assetTags | overview sidebar |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
Views
Atlan maps views from Google BigQuery to its View asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
DDL | definition | asset profile and overview sidebar |
Materialized views
Atlan maps materialized views from Google BigQuery to its MaterialisedView asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
TABLE_NAME | name | asset preview and profile, overview sidebar |
REMARKS | description | asset preview and profile, overview sidebar |
COLUMN_COUNT | columnCount | asset preview and profile, overview sidebar |
ROW_COUNT | rowCount | asset preview, profile, and filter, overview sidebar |
SIZE_BYTES | sizeBytes | asset filter and overview sidebar |
TABLE_TYPE | subType | asset preview and profile |
CREATED | sourceCreatedAt | asset profile and properties sidebar |
MODIFIED | sourceUpdatedAt | asset profile and properties sidebar |
OPTION_NAMES (require_partition_filter) | isPartitioned | API only |
DDL | definition | asset profile and overview sidebar |
Columns
Atlan supports nested columns up to level 15 for Google BigQuery to help you enrich your semi-structured data types. You can view nested columns in the Tree view or the asset sidebar of your table assets. Atlan also supports policy tagging on nested columns, so you can apply column-level security to fields at any nesting depth. Atlan maps columns from Google BigQuery to its Column asset type.
Atlan doesn't crawl primary key (PK) and foreign key (FK) information from Google BigQuery.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
COLUMN_NAME | name | asset preview and profile, overview sidebar |
REMARKS, DESCRIPTION | description | asset preview and profile, overview sidebar |
ORDINAL_POSITION | order | asset profile |
TYPE_NAME | dataType | asset preview, profile, and filter, overview sidebar |
IS_NULLABLE | isNullable | API only |
IS_PARTITIONING_COLUMN | isPartition | asset preview, profile, and filter |
CLUSTERING_COLUMN_LIST | isClustered | asset preview, profile, and filter |
Routines
Atlan maps routines (user-defined functions, table-valued functions and stored procedures) from Google BigQuery to its BigqueryRoutine asset type.
| Source property | Atlan property | Where in Atlan |
|---|---|---|
ROUTINE_NAME | name | asset preview and profile, overview sidebar and filter |
REMARKS | description | asset preview and profile, overview sidebar |
ROUTINE_DEFINITION | definition | asset profile and overview sidebar |
CREATED | sourceCreatedAt | asset profile and overview sidebar |
MODIFIED | sourceUpdatedAt | asset profile and overview sidebar |
ROUTINE_TYPE | BigqueryRoutineType | asset preview and profile, overview sidebar and filter |
ROUTINE_ARGUMENTS | BigqueryRoutineArguments | asset profile and overview sidebar |