Skip to main content

Query redaction for Google BigQuery

TL;DR

Reference for Redact SQL Text on Google BigQuery crawler and miner workflows: what Atlan redacts, what it preserves, and limitations to plan for.

Your AI can read this via Docs MCPInstall MCP →Connect

By default, Atlan stores the SQL text it extracts from Google BigQuery exactly as written at source. That text can contain literal values from your business data, such as email addresses in a WHERE clause or account numbers in a view definition.

Query redaction replaces those literal values with placeholders before Atlan stores them. Redaction runs in memory while the workflow extracts data, so the original values aren't written to Atlan's storage or sent outside your environment. Query structure, table references, and column references are preserved, so lineage is unaffected.

Redaction is off by default. You opt in per workflow using the Redact SQL Text option.

Enable redaction​

Redact SQL Text is a separate option on each workflow, so you can enable it for the crawler, the miner, or both. The miner doesn't inherit the crawler's setting, so enable it on each workflow whose SQL text you want redacted.

WorkflowWhere to find it
Google BigQuery crawlerConfigure crawler screen
Google BigQuery minerConfigure miner screen
Direct extraction only

Redact SQL Text is supported only for the Direct extraction method as of now. Workflows using the Agent extraction method don't show the option, so you can't turn redaction on from the workflow form on an agent-based connection. If you extract Google BigQuery through a Self Deployed Runtime and need redaction, contact Atlan support.

Changing this option applies to future runs only. See Limitations.

What Atlan redacts​

Redaction applies to the fields that carry SQL text for each workflow.

WorkflowRedacted content
CrawlerReplaces string and numeric literals in extracted SQL definitions with placeholders.
MinerQuery text from the INFORMATION_SCHEMA.JOBS_BY_PROJECT view

Within that text, Atlan replaces:

  • String literals with '<REDACTED:STRING>'.
  • Numeric literals with 1. Atlan substitutes a valid number rather than a text placeholder so that the redacted SQL remains parseable.

Atlan also removes comments from redacted SQL and reformats the statement, so stored text won't match the original layout.

Values Atlan preserves​

Some literals are safe to keep and can break the SQL if replaced, so redaction preserves them:

  • Date and time literals, such as '2026-01-01' or '2026-01-01 14:30:00'.
  • Month and weekday names, and date part keywords such as MONTH or QUARTER.
  • Currency codes, such as 'USD'.
  • Boolean words: TRUE, FALSE, YES, and NO.
  • Numbers in structural positions, such as LIMIT and OFFSET clauses, where the value controls the shape of the result rather than describing your data.

Everything else is treated as potentially sensitive and redacted.

Before and after​

A query extracted by the miner:

-- monthly EMEA revenue check
select order_id, customer_email
from `my-project.analytics.orders`
where region = 'EMEA' and order_total > 500.75
  and created_at >= '2026-01-01' and currency = 'USD'
limit 100;

The same query as Atlan stores it with Redact SQL Text enabled:

/* redacted: true, hash: 8382003735201380775, redacted_hash: -3200523710706585264 */
SELECT
  order_id,
  customer_email
FROM `my-project.analytics.orders`
WHERE
  region = '<REDACTED:STRING>'
  AND order_total > 1
  AND created_at >= '2026-01-01'
  AND currency = 'USD'
LIMIT 100

Note the following:

  • 'EMEA' was replaced, while the date and the currency code were preserved as allowlisted values.
  • 500.75 became 1. Redaction indicates that a number was present, not what it was.
  • LIMIT 100 was preserved, because the number controls the result size rather than describing your data.
  • The comment was removed and the statement was reformatted.
  • Atlan adds a header comment recording that the text was redacted, along with fingerprints of the original and redacted statements. The fingerprint values differ for every statement.
  • my-project.analytics.orders, order_id, and customer_email are unchanged, which is why lineage still resolves.

What redaction preserves​

Redaction only changes literal values. Because everything Atlan uses to build relationships stays intact, enabling redaction doesn't change:

Limitations​

Review these before you rely on redaction for a compliance requirement.

Redaction isn't retroactive. It applies only to SQL that Atlan extracts after you enable the option. SQL text already stored in Atlan from earlier runs stays unredacted. To remove it, work with Atlan support.

SQL that Atlan can't parse is stored unredacted. Redaction works by parsing each statement. When parsing fails, Atlan stores the original statement instead of dropping it. Dynamic SQL, statements wrapped in EXECUTE IMMEDIATE, and procedure bodies written in other languages are the most common cases. If your compliance requirement is that no unredacted SQL can be stored, treat this as a gap and raise it with Atlan support.

Redaction increases workflow runtime. Expect crawler and miner runs to take longer, and account for this if a workflow already runs close to its schedule interval.

Enabling redaction rewrites stored SQL. Because the stored text changes for every asset, the first run after you enable the option updates SQL text across the connection.

See also​