Mine queries through cloud object storage

||Connect docs via MCP

You can use Generic Miner to extract table- and column-level lineage from SQL query history that you have exported from any supported source system—without a live connection to that system. Generic Miner reads query logs stored in Atlan's own object store, Amazon S3, Google Cloud Storage (GCS), or Azure Blob Storage, and works with any connection you have already crawled in Atlan.

Structure query files

Generic Miner reads query history as line-delimited JSON (NDJSON). To prepare your files:

• Use a .json file extension.
• Store all files under a single prefix (folder path) in your chosen bucket or storage location.

For each file, verify that:

• Each line is a single JSON value. JSON objects can't be pretty-formatted or span multiple lines.
• Each SQL query appears on its own line.
• Lines aren't separated by commas.

Did you know?

You can include a default database, default schema, and session ID in each JSON record.

• If a SQL query references a table or view without a fully-qualified name, Atlan uses the default database and schema to resolve it.
• Including the session ID speeds up lineage processing. If you include it, make sure all queries belonging to the same session appear consecutively in the file.

The following example shows the JSON format. (Shown here split across multiple lines for readability—each record must be a single line in the actual file.)

{
  "QUERY_TEXT": "insert into NETFLIX_DB.PUBLIC.MOVIES_FILTERED as select m.* from MOVIES m where m.RATING > 5;",
  "DATABASE_NAME": "NETFLIX_DB",
  "SCHEMA_NAME": "PUBLIC",
  "SESSION_ID": "5c2f0a41-5d02-46f1-b9bd-ef80ad571013"
}

The JSON key names can be configured when you set up Generic Miner. In the example shown, DATABASE_NAME and SCHEMA_NAME are used to fully qualify the table MOVIES as NETFLIX_DB.PUBLIC.MOVIES. See Generate lineage with Generic Miner for details on configuring these keys.

Choose your input source

Generic Miner supports two input source types, which you select when configuring the workflow:

• Atlan Object Storage—files are staged in Atlan's own internal object store. You provide only a path prefix; no additional credentials are needed.
• External Object Storage—files are in your own bucket on AWS S3, Google Cloud Storage, or Azure Blob Storage. You configure access credentials directly in the Generic Miner workflow.

Set up Atlan object storage

To use Atlan's internal object store, upload your NDJSON query files to Atlan's storage bucket and note the path prefix where you placed the files. Raise a support request to get the staging path for your Atlan deployment.

When configuring Generic Miner, select Atlan Object Storage as the Manifest Source and enter the prefix path where your files are stored.

Set up external object storage

If your files are in your own bucket, complete the steps below for your cloud provider before configuring Generic Miner.

Amazon S3

Generic Miner accesses your S3 bucket by assuming an IAM role you create in your AWS account.

Step 1: Obtain Atlan's node instance role ARN

Contact Atlan support to request the Node Instance Role ARN for your Atlan EKS cluster. You need this value to configure the trust relationship in Step 3.

Step 2: Create IAM policy

1. In your AWS account, go to IAM → Policies → Create policy.
2. Select the JSON tab and paste the following, replacing <your-bucket> and <your-prefix> with your actual values:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AtlanGenericMinerReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket>",
        "arn:aws:s3:::<your-bucket>/<your-prefix>/*"
      ]
    }
  ]
}

3. Name the policy (for example, AtlanGenericMinerReadPolicy) and create it.

Step 3: Create IAM role with trust policy

1. In AWS, go to IAM → Roles → Create role.
2. Select Trusted entity type: AWS account → Another AWS account and enter the account ID from the Atlan node instance role ARN.
3. Attach the policy you created in Step 2.
4. Name the role (for example, AtlanGenericMinerRole) and create it.
5. Open the new role and click Edit trust policy. Replace the policy with:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<atlan-node-instance-role-arn>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Replace <atlan-node-instance-role-arn> with the Node Instance Role ARN from Step 1.

6. Copy the Role ARN from the role summary page (format: arn:aws:iam::123456789012:role/AtlanGenericMinerRole).

note

If your S3 bucket is encrypted with a customer-managed AWS KMS key, update the KMS key policy to grant the Atlan node instance role ARN the kms:Decrypt and kms:DescribeKey actions. Contact Atlan support for guidance.

warning

S3 buckets with VPC endpoints don't support cross-region requests. This may prevent Generic Miner from reading objects from your bucket.

Step 4: Configure in Generic Miner

In the Generic Miner workflow, under Manifest Source → External Object Storage, select AWS as the Storage Provider. Set Authentication to IAM Role and enter:

• AWS Role ARN: the Role ARN from Step 3
• Bucket Name: your S3 bucket name (no s3:// prefix)
• Prefix: path within the bucket where your NDJSON files are stored
• Region: your bucket's AWS region (for example, us-east-1)

Google Cloud Storage

Generic Miner authenticates to GCS using a service account with a JSON key.

Step 1: Create service account

1. In the Google Cloud Console, go to IAM & Admin → Service accounts.
2. Click Create service account and give it a name (for example, atlan-generic-miner).
3. Grant the service account the Storage Object Viewer role (roles/storage.objectViewer) on the GCS bucket containing your query history files.
4. Go to the service account's Keys tab → Add key → Create new key → JSON.
5. Download the JSON key file.

Step 2: Configure in Generic Miner

In the Generic Miner workflow, under Manifest Source → External Object Storage, select GCP as the Storage Provider. Set Authentication to Service Account and enter:

• Project ID: your GCP project ID
• Service Account JSON: paste the full contents of the JSON key file
• Bucket Name: your GCS bucket name
• Prefix: path within the bucket where your NDJSON files are stored

Azure Blob Storage

Generic Miner authenticates to Azure Blob Storage using a service principal.

Step 1: Register app and create client secret

1. In the Azure Portal, go to Microsoft Entra ID → App registrations → New registration.
2. Give the app a name (for example, atlan-generic-miner) and register it.
3. Note the Application (client) ID and Directory (tenant) ID from the overview page.
4. Go to Certificates & secrets → New client secret. Set an expiry and copy the secret Value immediately (shown only once).

Step 2: Grant access to storage container

1. In the Azure Portal, go to your Storage account → Containers → your container.
2. Click Access Control (IAM) → Add role assignment.
3. Assign the Storage Blob Data Reader role to the app registration created in Step 1.

Step 3: Configure in Generic Miner

In the Generic Miner workflow, under Manifest Source → External Object Storage, select AZURE as the Storage Provider. Set Authentication to Service Principal and enter:

• Tenant ID: Directory (tenant) ID from Step 1
• Client ID: Application (client) ID from Step 1
• Client Secret: the secret value from Step 1
• Account Name: your Azure Storage account name
• Container Name: the container holding your query history files
• Prefix: path within the container where your NDJSON files are stored

Configure generic miner

For step-by-step instructions on setting up the Generic Miner workflow in Atlan, see Generate lineage with Generic Miner.

Frequently asked questions

If I remove queries from object storage and re-run Generic Miner, does it remove lineage?

No. Atlan doesn't remove lineage from queries that are no longer present in the object store.

Does generic miner reprocess all files on every run?

Yes. Generic Miner processes all files at the configured prefix on each run and publishes any new lineage it finds. Remove older files when updating the files at the prefix to avoid reprocessing stale data.

Can I switch existing Generic Miner workflow to mine directly from source instead of object storage?

Yes. Edit the workflow configuration or set up a new workflow for the same connection. See Manage connectivity for details.

Are default database and schema keys always required?

The DATABASE_NAME and SCHEMA_NAME fields can be set to null if that information is already available in the queries. They serve as fallback values for queries that reference tables without a fully-qualified name.

Which SQL statements do I include?

Include DDL and DML statements such as CREATE, INSERT, UPDATE, DELETE, and MERGE. SELECT alone isn't required (it's a DQL statement and doesn't produce lineage on its own). Both UPDATE and DELETE can reference other tables, so include them to capture the full lineage graph.