Skip to main content
Community Hub
TL;DR

After you have crawled assets from Amazon Athena, mine its query history to construct lineage and retrieve usage and popularity metrics.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

Mine Amazon Athena Private Preview

Private preview

Mining query history from Amazon Athena is currently in private preview. To enable it for your workspace or share feedback, raise a support ticket or reach out to your Atlan customer success manager.

Use the Athena Miner workflow to read query history from Amazon Athena and build lineage and usage and popularity metrics for your crawled Athena assets. This page walks you through the prerequisites and through creating, configuring, and running the miner.

Prerequisites​

Before you begin:

  • Make sure you have crawled Amazon Athena at least once so a connection exists to mine.
  • Add the query history permissions to the IAM policy the crawler uses. See Add query history mining permissions.
  • Create the miner workflow while signed in to Atlan as a user who can publish to the connection. The miner publishes usage metrics as the user who created the workflow. A workflow created with an API token fails at the publish step with an ATLAS-403 error.
The most common miner failure is a missing grant

The crawler and the miner share one credential, so a working crawler doesn't mean the miner's grants are in place. athena:ListQueryExecutions and athena:BatchGetQueryExecution are in the base policy, but a policy scoped to specific workgroups must include every workgroup you mine. athena:ListWorkGroups is needed only when the Workgroups field is empty. A missing grant fails the miner at preflight. If your AWS admin won't grant the account-wide athena:ListWorkGroups, list the workgroups to mine in the Workgroups field and the miner never calls it.

Authentication​

The miner doesn't have its own credential form. It reuses the credential of the crawler connection you select, with the same IAM user or IAM role. Adding the miner is a policy update on that identity, not a new credential.

Create miner workflow​

To create the Athena miner workflow:

  1. In the top right of any screen, navigate to +New and then click New workflow.
  2. Under Marketplace, from the filters along the top, click Miner.
  3. From the list of packages, select Athena Miner and then click Setup Workflow.

Configure miner​

To configure the Athena miner:

  1. For Connection, select the connection to mine. (To select a connection, the crawler must have already run.)
  2. For Extraction method, choose your extraction method:
    • In Direct, Atlan reads query history from Athena directly.
    • In Self-Deployed Runtime, a Self-Deployed Runtime in your network reads query history. Select your runtime in the field that appears. This option might not be available on every workspace. Contact Atlan Support to enable it. For details on deploying the runtime, see:
  3. For Start date, choose how far back the first run reads. Athena keeps query history for 45 days, so the date is limited to the last 45 days. The default reads all 45 days. Later runs ignore this date for workgroups already mined and pick up from where the last successful run ended. A workgroup added to the miner later only gets lineage and usage for about the last four weeks of its history.
  4. For Workgroups, choose which workgroups to mine:
    • To mine specific workgroups, enter them as a JSON list, for example ["primary", "analytics"]. Names are case-sensitive. The miner doesn't call athena:ListWorkGroups in this mode.
    • To mine every enabled workgroup in the account and region, leave the default [] or clear the field. The miner discovers workgroups with athena:ListWorkGroups on each run.
  5. (Optional) For Excluded Workgroups, enter the exact workgroup names to leave out of popularity counts. Queries from these workgroups are still mined for lineage and SQL Intelligence.
  6. Under the preflight checks, click Check and confirm the checks pass. Preflight is required for the Direct extraction method. For what each check verifies, see Preflight checks.

Run miner​

To run the Athena miner, after completing the configuration:

  • To run the miner once, immediately, at the bottom of the screen, click the Run button.
  • To schedule the miner to run daily, at the bottom of the screen, click the Schedule & Run button.

Schedule the miner to run daily, a few hours after 00:00 UTC. Each run reads complete UTC days, up to 00:00 UTC of the day it runs. A query submitted before then that's still running when the miner runs is skipped and isn't picked up later. Run the miner at least once every four weeks. If more than about four weeks pass between successful runs, the oldest days aren't used for lineage or usage, even though Athena still keeps them.

After the miner completes, you see lineage for Amazon Athena assets based on query history, and usage and popularity metrics on the assets your queries read. Usage metrics can look higher for a few days after the first run, because the oldest backfilled queries are counted together on one day.

Preflight checks​

The miner runs these checks when you click Check. A failed authentication or permission check blocks the workflow. Throttling and timeouts show as failed rows but don't block the workflow.

CheckWhat it verifiesIf it fails
AWS authenticationAtlan can sign in with the connection's credential. For IAM role authentication, this includes assuming the role.Check the access key, or the role's trust policy and external ID.
WorkgroupsWith [] or an empty Workgroups field, that athena:ListWorkGroups is granted. With a workgroup list, the field is valid JSON and the check passes without calling AWS.Grant athena:ListWorkGroups, or list the workgroups to mine. A message that the field isn't a JSON list means the value needs fixing, for example ["primary"].
List query historyathena:ListQueryExecutions on each listed workgroup, or on up to 10 discovered workgroups.Grant the action on the workgroup named in the message. A "doesn't exist" message means the workgroup name is wrong. Names are case-sensitive.
Read query historyathena:BatchGetQueryExecution on the same workgroups, by reading back one query.Grant the action on the workgroup named in the message. A workgroup with no query history yet can't be read back, so the check shows a note instead of failing.

With the Self-Deployed Runtime extraction method, the form doesn't run preflight checks. The agent needs the same IAM permissions.

What the miner reads​

  • Successful queries only. Failed and canceled queries aren't mined. Queries that only read information_schema, and Atlan's own SELECT 1 connection test, are dropped.
  • Lineage comes from queries that write data, such as CREATE TABLE AS SELECT, INSERT INTO, and MERGE INTO. UPDATE statements on Iceberg tables don't produce lineage. Lineage only links assets that Atlan has already crawled, so re-run the crawler before mining when new tables or views appear.
  • Popularity counts read queries (SELECT) against your crawled assets.
  • Users. The Athena API doesn't return who ran a query. The miner reports each query's user as its workgroup, in the form athena:<workgroup>. Unique user counts and top users are per workgroup, not per person.
  • Cost is reported as data scanned in bytes, not as a currency amount.

Need help​

If you need help configuring the miner or setting up its permissions, contact Atlan Support by submitting a request.

See also​