## Generate lineage with Generic Miner

URL: https://docs.atlan.com/product/capabilities/lineage/how-tos/generate-lineage-with-generic-miner

> Configure and run Generic Miner to extract table-level and column-level lineage from SQL query history stored in Atlan's object store, Amazon S3, Google Cloud Storage, or Azure Blob Storage—without a live connection to the source system.

You can use **Generic Miner** to extract table-level and column-level lineage from SQL query history that you have exported from any supported source system. Generic Miner reads query logs stored in **Atlan's own object store** or an **external cloud bucket** (Amazon S3, Google Cloud Storage, or Azure Blob Storage), parses SQL statements using the dialect you choose, and attaches lineage to your existing Atlan connections—without a live connection to the source.

## When to use Generic Miner

Generic Miner is the fallback path for generating lineage when a native, direct-connect miner (running in SaaS or [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt)) isn't the right fit. Reach for it when:

- You don't want to use direct/SDR mode for a native connector, but you're willing to extract and maintain your own query-history export pipeline instead.
- You're generating lineage for a **custom connector**—one defined through the [App Framework](https://docs.atlan.com/llms/platform/build-apps/llms.txt), Atlan's APIs/SDK, or Asset Import—that has no native miner of its own. Generic Miner is the primary path to lineage for these connections.

## Prerequisites

Before you begin, make sure you have:

- Access to the **Generic Miner** app. Verify this by searching for **Generic Miner** in the Atlan marketplace. If you don't have access, [contact Atlan support](https://docs.atlan.com/support/submit-request) to request that the app be added to your tenant.
- A connection in Atlan for the source system whose queries you want to mine. Generic Miner always requires you to select a connection—this tells the workflow where to attach the resulting lineage Process assets. This includes:
 - Any of Atlan's native connectors (for example, Snowflake, BigQuery, or Redshift). A crawler must have already run for this connection.
 - A **custom connector** you've defined through the [App Framework](https://docs.atlan.com/llms/platform/build-apps/llms.txt), Atlan's APIs/SDK, or Asset Import. Any custom connector and connection that renders in Atlan is supported—a native Atlan connector for that source system isn't required.
- Query history files prepared and staged in Atlan's object store or your own cloud bucket. See [Mine queries through cloud object storage](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt) for instructions on formatting query files and configuring storage access.
- The JSON key names used in your query files for the SQL text, default database, default schema, and session ID fields. Refer to [Mine queries through cloud object storage](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt) for an example file structure.

<!--vale off-->

## Configure Generic Miner

<!--vale on-->

1. In your Atlan workspace, click **Connectors** in the left sidebar.

 If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**, search for **Generic Miner**, and select it.

3. Click **Install**. Once installation completes, click **Set up workflow** on the same tile.

4. For **Connection**, use the selector to choose the Atlan connection whose lineage you want to mine.

5. For **Parsing language**, select the SQL dialect that matches the source system that generated the queries. Generic Miner supports over 30 dialects, including ANSI SQL, Athena, Azure SQL, BigQuery, ClickHouse, Databricks, Hive, MySQL, Oracle, Postgres, Redshift, Snowflake, SQL Server, Teradata, Trino, and more.

Dialect names in this list follow the underlying SQL parser's naming convention, which doesn't always match a source system's common or marketing name. For example, Microsoft SQL Server is listed as **SQL Server**, not "MS SQL" or "MSSQL." If you don't see your source system by its common name, check for its formal engine name before assuming it's unsupported.

## Configure input source

Point Generic Miner to the location of your query history files using the **Manifest Source** selector.

### Atlan object storage

If your files are staged in Atlan's own object store:

1. Under **Manifest Source**, select **Atlan Object Storage**.

2. In the **Object Storage Prefix** field, enter the path within Atlan's internal bucket where your NDJSON files are stored (for example, `artifacts/apps/generic-miner/query-history`).

### External object storage

If your files are in your own AWS S3, GCP Cloud Storage, or Azure Blob Storage bucket:

1. Under **Manifest Source**, select **External Object Storage**. A credential configuration panel appears.

2. Under **Storage Provider**, select **AWS**, **GCP**, or **AZURE** based on where your files are stored.

3. Fill in the authentication and location fields for your provider:

**AWS (IAM Role)**
 - **AWS Role ARN**—ARN of the IAM role Generic Miner assumes to access your bucket. Leave blank to use Atlan's cluster IAM role (only if the bucket already grants access to Atlan's cluster role).
 - **External ID**—External ID required by the role's trust policy, if configured.
 - **Bucket Name**—S3 bucket name, without the `s3://` prefix.
 - **Prefix**—Path within the bucket where your NDJSON files are stored, without the bucket name or `s3://` scheme (for example, `query-history/2024`).
 - **Region**—AWS region of the bucket (for example, `us-east-1`).

**GCP (Service Account)**
 - **Project ID**—Your Google Cloud project ID.
 - **Service Account JSON**—JSON key contents of the service account that has read access to your GCS bucket.
 - **Bucket Name**—GCS bucket name.
 - **Prefix**—Path within the bucket where your NDJSON files are stored, without the `gs://` scheme.

**AZURE (Service Principal)**
 - **Tenant ID**—Azure Active Directory tenant ID.
 - **Client ID**—Application (client) ID of the app registration.
 - **Client Secret**—Client secret for the app registration.
 - **Account Name**—Azure Storage account name.
 - **Container Name**—Blob container name.
 - **Prefix**—Path within the container where your NDJSON files are stored.

## Map JSON keys for queries

Configure how Generic Miner reads individual fields from each JSON record so it can correctly interpret SQL statements and their execution context.

1. For **SQL JSON Key**, enter the name of the JSON key that contains the SQL text for each query. The default value is `QUERY_TEXT`.

 Use `QUERY_TEXT` if your JSON follows the structure described in [Mine queries through cloud object storage](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt). If your JSON uses a different field name (for example, `statement` or `query`), enter that key instead. The value must exactly match the key that contains the SQL string.

2. Under **Default Database JSON Key**, specify the JSON key that contains the default database name used when object names in the SQL aren't fully qualified. The default value is `DATABASE_NAME`.

 If your JSON stores the database name under a different key (for example, `default_db`), enter that key instead. This value helps Generic Miner correctly resolve table and view references when object names aren't fully qualified.

3. In **Default Schema JSON Key**, provide the JSON key that specifies the default schema name. The default value is `SCHEMA_NAME`. This value helps Generic Miner qualify object names when the schema isn't explicitly specified in the SQL.

4. Use **Session ID JSON Key** to point to the JSON key that contains the session identifier for the query. The default value is `SESSION_ID`.

 Grouping queries by session improves lineage accuracy for related or sequential statements. This example shows a JSON record using the default key names:

 ```json
 {
 "QUERY_TEXT": "insert into NETFLIX_DB.PUBLIC.MOVIES_FILTERED as select m.* from MOVIES m where m.RATING > 5;",
 "DATABASE_NAME": "NETFLIX_DB",
 "SCHEMA_NAME": "PUBLIC",
 "SESSION_ID": "5c2f0a41-5d02-46f1-b9bd-ef80ad571013"
 }
 ```

5. Optionally, for **Start Time JSON Key**, enter the JSON key that contains the per-query execution timestamp as epoch milliseconds in UTC (for example, `1748606400000`). If you leave this blank, or if individual records lack the value, Generic Miner synthesizes a deterministic timestamp based on file and row order.

## Run and verify workflow

1. After completing the configuration, click **Run** to start a one-time Generic Miner run, or click **Schedule & Run** to set up recurring runs on an hourly, daily, weekly, or monthly cadence.

2. Monitor the workflow status from the workflow run page and review the logs to confirm the run completed successfully.

3. To verify lineage, open one of the tables or views in Atlan that were covered by the mined queries and navigate to the **Lineage** tab. Confirm that the upstream and downstream relationships reflect the queries processed by Generic Miner.

## See also

- [Mine queries through cloud object storage](https://docs.atlan.com/llms/catalog/connector-capabilities/mine-queries-through-cloud-object-storage/llms.txt): Prepare query history files and configure storage access for Atlan Object Storage, Amazon S3, Google Cloud Storage, or Azure Blob Storage.
- [View lineage](https://docs.atlan.com/llms/catalog/lineage/view-lineage/llms.txt): Learn how to explore the lineage created by Generic Miner.

---