Skip to main content

Generate lineage for ClickHouse assets
App

To generate lineage for your ClickHouse assets, the ClickHouse Miner reads query history from system.query_log. Before you run the miner, query logging must be enabled and the crawler user must have access to that table. Once that's in place, you can run the miner to start building lineage.

Before you begin​

Make sure you have:

  • Set up ClickHouse by following Set up ClickHouse.
  • Crawled ClickHouse metadata by following Crawl ClickHouse.
  • ClickHouse 21.8 or later (recommended for full system.query_log support).

Generate lineage​

Validate your ClickHouse environment so the miner can extract query history from system.query_log.

  1. Confirm that query logging is enabled. ClickHouse enables the query_log table by default. Verify by running:

    SELECT count(*) FROM system.query_log;

    If this query returns an error, query logging may be disabled. Check your ClickHouse server configuration (config.xml or config.d/) and make sure the <query_log> section is present:

    <query_log>
        <database>system</database>
        <table>query_log</table>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>

  2. Grant the crawler user access to read from system.query_log.

    GRANT SELECT ON system.query_log TO {{username}};

  3. Verify that system.query_log is collecting data. Run a sample query and then check that it appears in the log:

    SELECT 1;
    SYSTEM FLUSH LOGS;
    SELECT count(*) FROM system.query_log WHERE event_date = today();

  4. To limit the query log retention period, configure query_log TTL settings in your server configuration. The default TTL is sufficient for most deployments.

Run miner​

After you confirm query logging is enabled, create and run the ClickHouse Miner workflow to extract query history and generate lineage:

  1. Follow Mine ClickHouse to select the ClickHouse Miner package and configure connection and extraction method.
  2. Run the miner once or on a schedule.

Once the miner completes, your existing ClickHouse assets are enriched with lineage from query history in system.query_log.

Need help​

If you need help configuring the miner or enabling query logging, contact Atlan Support by submitting a request.