Skip to main content
Community Hub

Crawl Confluent Kafka

TL;DR

Configure and run the Confluent Kafka crawler to catalog topics, consumer groups, and schemas, collect topic sizes, and build lineage from Kafka Connect connectors to Snowflake tables.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

Extract metadata assets from your Confluent Kafka cluster into Atlan. After you set up Confluent Kafka, you can crawl topics, consumer groups, and schema information, collect topic sizes, and build lineage from Kafka Connect connectors to the Snowflake tables they write to.

Prerequisites​

Before you begin, complete the following prerequisites:

  • Confluent Kafka setup: You've configured the Confluent Kafka permissions needed for Atlan to connect to your cluster.
  • Cloud API key (for topic sizes or Connect lineage): You've created a Cloud API key with the roles your features need, and you have your cluster ID.
  • Snowflake connection (for Connect lineage): You've crawled the Snowflake account that your Snowflake sink connectors write to. Atlan links lineage only to Snowflake tables that already exist in Atlan.
  • Order of operations: Review the order of operations to understand the sequence of tasks for crawling metadata.
  • Access to Atlan workspace: You have the required permissions in Atlan to create and manage a connection.

Create crawler workflow​

  1. In your Atlan workspace, click Connectors in the left sidebar.

    • If you are using the Old UI (Classic), click New Workflow in the top navigation.

  2. Click Marketplace.

  3. Search for Confluent Kafka Assets and select it.

  4. Click Install.

  5. Once installation completes, click Setup Workflow on the same tile.

Configure extraction​

Select your extraction method and provide the connection details for your Confluent Kafka cluster.

Offline extraction sunset

The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the Agent extraction method with Self-Deployed Runtime.

Atlan connects directly to your Confluent Kafka cluster and crawls metadata over the network.

  1. For Bootstrap servers, enter the hostnames of your Confluent Kafka brokers. Separate multiple hostnames with a comma , or semicolon ;.

  2. For API Key and API Secret, enter the resource-specific API key and secret you created.

  3. For Security protocol, keep the default SASL_SSL to connect over a Secure Sockets Layer (SSL) channel, or select SASL_PLAINTEXT to connect over a non-encrypted channel.

  4. To collect topic sizes or build Connect lineage, set Enable Cloud API to True and provide the following details:

    • For Cloud API Key and Cloud API Secret, enter the Cloud API key and secret you created. Don't reuse the resource-specific API key from the previous step: the Confluent Cloud APIs reject it.
    • For Cluster ID, enter the ID of your cluster, for example lkc-xxxxx.

  5. To crawl Schema Registry subjects alongside Kafka, set Include Schema Registry to True and provide the following details:

    • For Schema registry host, enter the URL of your Schema Registry endpoint (for example, https://psrc-xxxxx.us-east-2.aws.confluent.cloud).
    • For API Key and API Secret, enter the Schema Registry API key and secret you created.

  6. Click Test Authentication to confirm connectivity, then click Next.

Configure connection​

Set up the connection identity and access controls for your Confluent Kafka source.

  1. Provide a Connection Name that represents your source environment. For example, production, development, gold, or analytics.

  2. Under Connection Admins, add the users or groups that can manage this connection. If you leave this empty, no one can manage the connection, including admins.

  3. At the bottom of the screen, click Next.

Configure crawling options​

On the Metadata page, you can override the defaults for any of these options. If an asset appears in both include and exclude filters, the exclude filter takes precedence. When Schema Registry is enabled, the topic include and exclude regex also apply to schema subjects. Subjects are matched using their base topic name, without the -key or -value suffix.

  • Click Exclude topic regex to exclude specific topics. Defaults to no exclusions if none are specified.
  • Click Include topic regex to limit crawling to specific topics. Defaults to all topics if none are specified.
  • For Include Cloud Metrics, select True to collect topic sizes from the Confluent Cloud Metrics API.
  • For Include Connect Lineage, select True to build lineage from your Kafka Connect connectors to the Snowflake tables they write to. Lineage covers only the topics that the regex filters include. If you later turn this option off, the next run removes the Connect lineage it created.
  • For Skip internal topics, keep the default Yes to skip internal Kafka topics, or select No to crawl them.

Include Cloud Metrics and Include Connect Lineage are hidden when Enable Cloud API is False in the credential.

Connections that collected topic sizes before Enable Cloud API

Existing connections that already collected topic sizes keep doing so without any change. If you edit and save the credential of such a connection, set Enable Cloud API to True and confirm the Cloud API Key, Cloud API Secret, and Cluster ID, then set Include Cloud Metrics to True in the workflow. Otherwise, topic sizes are no longer collected.

Run crawler​

After configuring all options, run or schedule the crawler.

  1. For Direct extraction, click Preflight checks to validate permissions and configuration before running. For Agent extraction, skip this step.

  2. Click Run to run the crawler once immediately, or click Schedule & Run to schedule the crawler to run hourly, daily, weekly, or monthly.

Once the crawler completes, the assets appear on Atlan's asset page. With Include Connect Lineage turned on, lineage appears on the Kafka topics and the Snowflake tables they feed. If Include Connect Lineage is on but the credential has no usable Cloud API configuration, or Atlan can't read your connectors, the workflow fails instead of completing, so lineage from earlier runs isn't removed.

See also​