Skip to main content

Generate lineage with Generic Miner

Connect docs via MCP

You can use Generic Miner to extract table-level and column-level lineage from SQL query history that you have exported from any supported source system. Generic Miner reads query logs stored in Atlan's own object store or an external cloud bucket (Amazon S3, Google Cloud Storage, or Azure Blob Storage), parses SQL statements using the dialect you choose, and attaches lineage to your existing Atlan connections—without a live connection to the source.

Prerequisites​

Before you begin, make sure you have:

  • Access to the Generic Miner app. Verify this by searching for Generic Miner in the Atlan marketplace. If you don't have access, contact Atlan support to request that the app be added to your tenant.
  • A connection in Atlan for the source system whose queries you want to mine (for example, Snowflake, BigQuery, Redshift, or any other supported connector). A crawler must have already run for this connection.
  • Query history files prepared and staged in Atlan's object store or your own cloud bucket. See Mine queries through cloud object storage for instructions on formatting query files and configuring storage access.
  • The JSON key names used in your query files for the SQL text, default database, default schema, and session ID fields. Refer to Mine queries through cloud object storage for an example file structure.

Configure Generic Miner​

  1. In your Atlan workspace, click Connectors in the left sidebar.

    If you are using the Old UI (Classic), click New Workflow in the top navigation.

  2. Click Marketplace, search for Generic Miner, and select it.

  3. Click Install. Once installation completes, click Set up workflow on the same tile.

  4. For Connection, use the selector to choose the Atlan connection whose lineage you want to mine.

  5. For Parsing language, select the SQL dialect that matches the source system that generated the queries. Generic Miner supports over 30 dialects, including ANSI SQL, Athena, Azure SQL, BigQuery, ClickHouse, Databricks, Hive, MySQL, Oracle, Postgres, Redshift, Snowflake, SQL Server, Teradata, Trino, and more.

Configure input source​

Point Generic Miner to the location of your query history files using the Manifest Source selector.

Atlan object storage​

If your files are staged in Atlan's own object store:

  1. Under Manifest Source, select Atlan Object Storage.

  2. In the Object Storage Prefix field, enter the path within Atlan's internal bucket where your NDJSON files are stored (for example, artifacts/apps/generic-miner/query-history).

External object storage​

If your files are in your own AWS S3, GCP Cloud Storage, or Azure Blob Storage bucket:

  1. Under Manifest Source, select External Object Storage. A credential configuration panel appears.

  2. Under Storage Provider, select AWS, GCP, or AZURE based on where your files are stored.

  3. Fill in the authentication and location fields for your provider:

AWS (IAM Role)

  • AWS Role ARN—ARN of the IAM role Generic Miner assumes to access your bucket. Leave blank to use Atlan's cluster IAM role (only if the bucket already grants access to Atlan's cluster role).
  • External ID—External ID required by the role's trust policy, if configured.
  • Bucket Name—S3 bucket name, without the s3:// prefix.
  • Prefix—Path within the bucket where your NDJSON files are stored, without the bucket name or s3:// scheme (for example, query-history/2024).
  • Region—AWS region of the bucket (for example, us-east-1).

GCP (Service Account)

  • Project ID—Your Google Cloud project ID.
  • Service Account JSON—JSON key contents of the service account that has read access to your GCS bucket.
  • Bucket Name—GCS bucket name.
  • Prefix—Path within the bucket where your NDJSON files are stored, without the gs:// scheme.

AZURE (Service Principal)

  • Tenant ID—Azure Active Directory tenant ID.
  • Client ID—Application (client) ID of the app registration.
  • Client Secret—Client secret for the app registration.
  • Account Name—Azure Storage account name.
  • Container Name—Blob container name.
  • Prefix—Path within the container where your NDJSON files are stored.

Map JSON keys for queries​

Configure how Generic Miner reads individual fields from each JSON record so it can correctly interpret SQL statements and their execution context.

  1. For SQL JSON Key, enter the name of the JSON key that contains the SQL text for each query. The default value is QUERY_TEXT.

    Use QUERY_TEXT if your JSON follows the structure described in Mine queries through cloud object storage. If your JSON uses a different field name (for example, statement or query), enter that key instead. The value must exactly match the key that contains the SQL string.

  2. Under Default Database JSON Key, specify the JSON key that contains the default database name used when object names in the SQL aren't fully qualified. The default value is DATABASE_NAME.

    If your JSON stores the database name under a different key (for example, default_db), enter that key instead. This value helps Generic Miner correctly resolve table and view references when object names aren't fully qualified.

  3. In Default Schema JSON Key, provide the JSON key that specifies the default schema name. The default value is SCHEMA_NAME. This value helps Generic Miner qualify object names when the schema isn't explicitly specified in the SQL.

  4. Use Session ID JSON Key to point to the JSON key that contains the session identifier for the query. The default value is SESSION_ID.

    Grouping queries by session improves lineage accuracy for related or sequential statements. This example shows a JSON record using the default key names:

    {
      "QUERY_TEXT": "insert into NETFLIX_DB.PUBLIC.MOVIES_FILTERED as select m.* from MOVIES m where m.RATING > 5;",
      "DATABASE_NAME": "NETFLIX_DB",
      "SCHEMA_NAME": "PUBLIC",
      "SESSION_ID": "5c2f0a41-5d02-46f1-b9bd-ef80ad571013"
    }

  5. Optionally, for Start Time JSON Key, enter the JSON key that contains the per-query execution timestamp as epoch milliseconds in UTC (for example, 1748606400000). If you leave this blank, or if individual records lack the value, Generic Miner synthesizes a deterministic timestamp based on file and row order.

Run and verify workflow​

  1. After completing the configuration, click Run to start a one-time Generic Miner run, or click Schedule & Run to set up recurring runs on an hourly, daily, weekly, or monthly cadence.

  2. Monitor the workflow status from the workflow run page and review the logs to confirm the run completed successfully.

  3. To verify lineage, open one of the tables or views in Atlan that were covered by the mined queries and navigate to the Lineage tab. Confirm that the upstream and downstream relationships reflect the queries processed by Generic Miner.

See also​