## Compute and cost tracking

URL: https://docs.atlan.com/product/capabilities/governance/data-quality/references/compute-and-cost-tracking

> Understand how Data Quality Studio uses compute resources, how costs are calculated, and practical ways to track and optimize spend for Snowflake, Databricks, and BigQuery.

Data Quality Studio (DQS) uses platform compute resources to manage rule definitions and results, and run data quality checks on monitored tables. This document explains how DQS uses compute in Snowflake, Databricks, and BigQuery, how to track costs for each compute type, and ways to reduce compute costs.

## Types of compute

DQS uses different compute resources depending on your platform. Each compute type serves specific functions in the data quality workflow.

### Snowflake

Atlan DQS uses three types of Snowflake compute:

### Databricks

Atlan DQS uses two types of Databricks compute:

### BigQuery

Atlan DQS uses two types of BigQuery compute:

## Track compute cost

Track compute costs using platform-specific billing views and tables. Each platform provides different methods to query usage and calculate spend.

### Snowflake

Use Snowflake's `ACCOUNT_USAGE` views to track credits spent for running DQS. Your user needs access to the `SNOWFLAKE.ACCOUNT_USAGE` schema before running queries. Adjust time filters and database names as needed for your environment.

- **Snowflake managed DMF compute**

 - Overall cost:

```sql
SELECT
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.DATA_QUALITY_MONITORING_USAGE_HISTORY;
```

 - Cost per table:

```sql
SELECT
 CONCAT(DATABASE_NAME, '.', SCHEMA_NAME, '.', TABLE_NAME) AS ENTITY,
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.DATA_QUALITY_MONITORING_USAGE_HISTORY
GROUP BY ENTITY
ORDER BY TOTAL_CREDITS DESC;
```

 - Daily trend:

```sql
SELECT
 CONCAT(DATABASE_NAME, '.', SCHEMA_NAME, '.', TABLE_NAME) AS ENTITY,
 TO_DATE(START_TIME) AS USAGE_DATE,
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.DATA_QUALITY_MONITORING_USAGE_HISTORY
GROUP BY USAGE_DATE, ENTITY
ORDER BY USAGE_DATE DESC;
```

- **Warehouse**

 Replace `<your_warehouse_name>` with the name of your DQ warehouse (for example, `COMPUTE_WH`).

 - Overall cost:

```sql
SELECT
 NAME,
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.METERING_HISTORY
WHERE SERVICE_TYPE = 'WAREHOUSE_METERING'
 AND NAME = '<your_warehouse_name>'
GROUP BY NAME;
```

 - Daily trend:

```sql
SELECT
 NAME,
 TO_DATE(START_TIME) AS USAGE_DATE,
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.METERING_HISTORY
WHERE SERVICE_TYPE = 'WAREHOUSE_METERING'
 AND NAME = '<your_warehouse_name>'
GROUP BY NAME, USAGE_DATE
ORDER BY USAGE_DATE DESC;
```

- **Serverless task**

 Replace `<your_dq_database>` with the name of your DQ database (for example, `ATLAN_DQ_DQ_DEV`).

 - Overall cost:

```sql
SELECT
 CONCAT(DATABASE_NAME, '.', SCHEMA_NAME, '.', TASK_NAME) AS TASK,
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.SERVERLESS_TASK_HISTORY
WHERE DATABASE_NAME LIKE '<your_dq_database>'
GROUP BY TASK;
```

 - Daily trend:

```sql
SELECT
 CONCAT(DATABASE_NAME, '.', SCHEMA_NAME, '.', TASK_NAME) AS TASK,
 TO_DATE(START_TIME) AS USAGE_DATE,
 SUM(CREDITS_USED) AS TOTAL_CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.SERVERLESS_TASK_HISTORY
WHERE DATABASE_NAME LIKE '<your_dq_database>'
GROUP BY USAGE_DATE, TASK
ORDER BY USAGE_DATE DESC;
```

### Databricks

Use the `system.billing.usage` table to track DBU spent for running DQS. Your user or service principal needs SELECT permission to query `system.billing.usage` and `system.billing.list_prices`. Adjust time filters as needed for your environment.

- **Jobs compute**

 Track DBU consumption and cost for all DQ jobs:

 - Overall cost:

```sql
SELECT
 COUNT(*) AS no_of_dq_runs,
 SUM(u.usage_quantity) AS total_dbus,
 SUM(u.usage_quantity * p.pricing.default) AS total_cost_usd
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
 ON u.cloud = p.cloud
 AND u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_end_time <= p.price_end_time)
WHERE u.usage_metadata.job_name LIKE 'DQ_RUN_%'
 AND u.billing_origin_product = 'JOBS';
```

 - Cost per monitored table:

```sql
SELECT
 REGEXP_EXTRACT(u.usage_metadata.job_name, '([^/]+/[^/]+/[^/]+$)', 1) AS table_name,
 COUNT(DISTINCT u.usage_metadata.job_run_id) AS no_of_dq_runs,
 SUM(u.usage_quantity) AS total_dbus,
 SUM(u.usage_quantity * p.pricing.default) AS total_cost_usd
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
 ON u.cloud = p.cloud
 AND u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_end_time <= p.price_end_time)
WHERE u.usage_metadata.job_name LIKE 'DQ_RUN_%'
 AND u.billing_origin_product = 'JOBS'
GROUP BY u.usage_metadata.job_name
ORDER BY total_cost_usd DESC;
```

 - Daily trend:

```sql
SELECT
 DATE_TRUNC('day', u.usage_start_time) AS day_start,
 COUNT(DISTINCT u.usage_metadata.job_run_id) AS no_of_dq_runs,
 SUM(u.usage_quantity) AS total_dbus,
 SUM(u.usage_quantity * p.pricing.default) AS total_cost_usd
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
 ON u.cloud = p.cloud
 AND u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_end_time <= p.price_end_time)
WHERE u.usage_metadata.job_name LIKE 'DQ_RUN_%'
 AND u.billing_origin_product = 'JOBS'
GROUP BY DATE_TRUNC('day', u.usage_start_time)
ORDER BY day_start;
```

- **SQL warehouses**

 Replace `<warehouse_id>` with the ID of the SQL warehouse you have chosen for DQS.

 - Overall cost:

```sql
SELECT
 SUM(u.usage_quantity) AS total_dbus,
 SUM(u.usage_quantity * p.pricing.default) AS total_cost_usd
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
 ON u.cloud = p.cloud
 AND u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_end_time <= p.price_end_time)
WHERE u.usage_metadata.warehouse_id = '<warehouse_id>'
 AND u.billing_origin_product = 'SQL';
```

 - Daily trend:

```sql
SELECT
 DATE_TRUNC('day', u.usage_start_time) AS day_start,
 SUM(u.usage_quantity) AS total_dbus,
 SUM(u.usage_quantity * p.pricing.default) AS total_cost_usd
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
 ON u.cloud = p.cloud
 AND u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_end_time <= p.price_end_time)
WHERE u.usage_metadata.warehouse_id = '<warehouse_id>'
 AND u.billing_origin_product = 'SQL'
GROUP BY DATE_TRUNC('day', u.usage_start_time)
ORDER BY day_start;
```

### BigQuery

Use BigQuery's `INFORMATION_SCHEMA.JOBS_BY_PROJECT` view to track slot consumption and bytes processed for DQS. Your user or service account needs the `roles/bigquery.resourceViewer` role (or equivalent `bigquery.jobs.listAll` permission) on the project. Replace the following placeholders before running the queries:

- `<your-project-id>`: The GCP project where DQS is configured.
- `<region>`: The BigQuery region (for example, `us` or `eu`).
- `<your-on-demand-price>`: Your on-demand rate per TiB (for example, `6.25` USD/TiB).
- `<your-dqs-service-account-email>`: The service account you configured for DQS, for example `atlan-dq-service-account@<your-project-id>.iam.gserviceaccount.com`. If you used a different name when [setting up BigQuery](https://docs.atlan.com/llms/governance/data-quality/set-up-bigquery/llms.txt), substitute that email instead.

DQS jobs are attributed to the service account that Atlan uses to run scheduled queries. All queries below filter on this service account so only DQS usage is counted.

- **BigQuery slots**

 - Overall cost (on-demand):

```sql
SELECT
 COUNT(*) AS dq_job_count,
 SUM(total_bytes_billed) / POW(2, 40) AS total_tib_billed,
 (SUM(total_bytes_billed) / POW(2, 40)) * <your-on-demand-price> AS total_cost_usd
FROM `<your-project-id>`.`region-<region>`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE user_email = '<your-dqs-service-account-email>'
 AND job_type = 'QUERY'
 AND state = 'DONE';
```

 - Overall cost (reservations or editions):

```sql
SELECT
 COUNT(*) AS dq_job_count,
 SUM(total_slot_ms) / 1000 AS total_slot_seconds,
 SUM(total_slot_ms) / (1000 * 60 * 60) AS total_slot_hours
FROM `<your-project-id>`.`region-<region>`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE user_email = '<your-dqs-service-account-email>'
 AND job_type = 'QUERY'
 AND state = 'DONE';
```

 Multiply `total_slot_hours` by your reservation or edition slot-hour rate to get cost.

 - Cost per monitored table:

```sql
SELECT
 CONCAT(ref.project_id, '.', ref.dataset_id, '.', ref.table_id) AS monitored_table,
 COUNT(*) AS dq_job_count,
 SUM(total_bytes_billed) / POW(2, 40) AS total_tib_billed,
 (SUM(total_bytes_billed) / POW(2, 40)) * <your-on-demand-price> AS total_cost_usd
FROM `<your-project-id>`.`region-<region>`.INFORMATION_SCHEMA.JOBS_BY_PROJECT,
 UNNEST(referenced_tables) AS ref
WHERE user_email = '<your-dqs-service-account-email>'
 AND job_type = 'QUERY'
 AND state = 'DONE'
 AND ref.dataset_id NOT IN ('atlan_dq')
GROUP BY monitored_table
ORDER BY total_cost_usd DESC;
```

 The `ref.dataset_id NOT IN (...)` filter excludes the DQS dataset so only monitored tables appear. The default DQS dataset is `atlan_dq`, but you can rename it during [setup](https://docs.atlan.com/llms/governance/data-quality/set-up-bigquery/llms.txt). If you have multiple Atlan connections in the same GCP project, add all corresponding DQS datasets to the NOT IN clause. For example, `NOT IN ('atlan_dq', 'atlan_dq_sales', 'atlan_dq_marketing')`.

 - Daily trend:

```sql
SELECT
 DATE(creation_time) AS usage_date,
 COUNT(*) AS dq_job_count,
 SUM(total_bytes_billed) / POW(2, 40) AS total_tib_billed,
 (SUM(total_bytes_billed) / POW(2, 40)) * <your-on-demand-price> AS total_cost_usd
FROM `<your-project-id>`.`region-<region>`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE user_email = '<your-dqs-service-account-email>'
 AND job_type = 'QUERY'
 AND state = 'DONE'
GROUP BY usage_date
ORDER BY usage_date DESC;
```

To aggregate across all projects in an organization, query `region-<region>.INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION` instead and add a `project_id` column to the `GROUP BY`.

## Reduce compute cost

Follow these practices to keep DQS compute costs efficient and predictable.

### Snowflake

- Start small and scale gradually
 - Start with an X-Small warehouse for DQS control-plane operations.
 - Scale up only if you see consistent performance bottlenecks.
- Configure auto suspend policies on warehouses
 - Set aggressive auto suspend timeouts to avoid paying for idle warehouse time.
- Schedule DQ runs thoughtfully
 - Run critical tables hourly or daily based on business SLAs.
 - Run non-critical tables weekly or after upstream ETL jobs finish.
- Use incremental DQ monitoring to reduce scanned volume.
- Delete low-value rules that consistently pass or are no longer critical.

Regularly reviewing warehouse metering and usage helps you keep DQS costs transparent and under control as your coverage grows.

### Databricks

- Start small and scale gradually
 - Start with a 2X-Small serverless SQL warehouse for DQS control-plane operations.
 - Increase size only if you see consistent performance bottlenecks.
- Configure auto stop policies on SQL warehouses
 - Set aggressive auto stop timeouts to avoid paying for idle warehouse time.
 - Prefer serverless warehouses where available for faster startup times.
- Schedule DQ runs thoughtfully
 - Run critical tables hourly or daily based on business SLAs.
 - Run non-critical tables weekly or after upstream ETL jobs finish.
- Use incremental DQ monitoring to reduce scanned volume.
- Delete low-value rules that consistently pass or are no longer critical.

Regularly reviewing jobs compute and SQL warehouse usage helps you keep DQS costs transparent and under control as your coverage grows.

### BigQuery

- Partition and cluster monitored tables
 - DQS rules scan only the columns they need, so clustering and partitioning on frequently filtered columns dramatically reduce bytes processed per run.
- Schedule DQ runs thoughtfully
 - Run critical tables hourly or daily based on business SLAs.
 - Run non-critical tables weekly or after upstream ETL jobs finish.
- Match your pricing model to workload shape
 - Use on-demand pricing for bursty or low-volume DQ workloads.
 - Consider reservations or editions (with autoscaler) if DQ becomes a sustained, high-volume workload.
- Delete low-value rules that consistently pass or are no longer critical.
- Monitor the DQS service account in `INFORMATION_SCHEMA.JOBS_BY_PROJECT`
 - Review top-cost tables and rules weekly and retire ones that no longer justify their spend.

Regularly reviewing slot consumption and bytes billed for the DQS service account helps you keep DQS costs transparent and under control as your coverage grows.

## See also

* [Operations](https://docs.atlan.com/llms/governance/data-quality/operations/llms.txt): Learn about data quality operations in Snowflake
* [Data quality permissions](https://docs.atlan.com/llms/governance/data-quality/data-quality-permissions/llms.txt): Understand permission scopes for data quality
* [Set up Snowflake](https://docs.atlan.com/llms/governance/data-quality/set-up-snowflake/llms.txt): Configure Snowflake for data quality monitoring
* [Set up Databricks](https://docs.atlan.com/llms/governance/data-quality/set-up-databricks/llms.txt): Configure Databricks for data quality monitoring
* [Set up BigQuery](https://docs.atlan.com/llms/governance/data-quality/set-up-bigquery/llms.txt): Configure BigQuery for data quality monitoring

---