## Crawl Apache Kafka

URL: https://docs.atlan.com/apps/connectors/messaging/apache-kafka/how-tos/crawl-apache-kafka

> Crawl metadata from Apache Kafka to catalog topics, consumer groups, and clusters. Extract schema information from Confluent Schema Registry after configuring cluster permissions.

Extract metadata assets from your [Apache Kafka](https://docs.atlan.com/llms/connectors/apache-kafka/set-up-apache-kafka/llms.txt) cluster into Atlan. After configuring the necessary cluster permissions, you can crawl topics, consumer groups, clusters, and optionally schema information from [Confluent Schema Registry](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt). Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting.

## Prerequisites

Before you begin, complete the following prerequisites:

- **Apache Kafka setup:** You've [configured the Apache Kafka permissions](https://docs.atlan.com/llms/connectors/apache-kafka/set-up-apache-kafka/llms.txt) needed for Atlan to connect to your cluster.
- **Schema Registry setup (if crawling schemas):** You've completed the [Confluent Schema Registry setup](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt) and have the Schema Registry endpoint, API key, and API secret ready.
- **Order of operations:** Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) to understand the sequence of tasks for crawling metadata.
- **Access to Atlan workspace:** You have the required permissions in Atlan to create and manage a connection.

## Create crawler workflow

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Apache Kafka Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Configure extraction

Select your extraction method and provide the connection details for your Apache Kafka cluster.

:::warning Offline extraction sunset
The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt).
:::

### Direct

Atlan connects directly to your Apache Kafka cluster and crawls metadata over the network.

1. For _Bootstrap servers_, enter one or more hostnames of your Apache Kafka brokers. For multiple hostnames, separate each entry with a comma `,` or semicolon `;`.

2. For _Authentication_, choose the method that matches your cluster configuration:

 - **No Auth** -- select this if your cluster doesn't require authentication.
 - **Basic** -- enter the [username and password](https://docs.atlan.com/llms/connectors/apache-kafka/set-up-apache-kafka/llms.txt) configured for Atlan using SASL/PLAIN.
 - **SCRAM** -- enter the [username and password](https://docs.atlan.com/llms/connectors/apache-kafka/set-up-apache-kafka/llms.txt) and choose the SCRAM mechanism (SCRAM-SHA-256 or SCRAM-SHA-512).
 - **mTLS** -- upload the client certificate and private key for mutual TLS authentication.

3. For _Security protocol_, select **Plaintext** or **SSL** for No Auth, and **SASL\_PLAINTEXT** or **SASL\_SSL** for Basic and SCRAM authentication.

4. To crawl Schema Registry subjects alongside Kafka, set _Include Schema Registry_ to **True** and provide the following details:
 - For _Schema registry host_, enter the URL of your Schema Registry endpoint (for example, `https://psrc-xxxxx.us-east-2.aws.confluent.cloud`).
 - For _API Key_, enter the [Schema Registry API key](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt) you created.
 - For _API Secret_, enter the [Schema Registry API secret](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt) you created.
5. Click **Test Authentication** to confirm connectivity, then click **Next**.

### Agent

[Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt) executes metadata extraction within your organization's environment, keeping all connections inside your network perimeter.

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
2. Confirm the runtime can reach your Apache Kafka cluster over your local network and that [network security](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-network-security/llms.txt) is configured.

3. Under _Secure Agent Configuration_, select your deployed agent from the **Agent** dropdown and the secret store from the **Secret Store** dropdown.

4. For _Bootstrap servers_, enter one or more hostnames of your Apache Kafka brokers as reachable from within your network.

5. For _Authentication_, choose the method that matches your cluster configuration:

 - **No Auth** -- select this if your cluster doesn't require authentication.
 - **Basic** -- reference the secret store path for the username and password configured for Atlan using SASL/PLAIN.
 - **SCRAM** -- reference the secret store path for the username and password and choose the SCRAM mechanism (SCRAM-SHA-256 or SCRAM-SHA-512).
 - **mTLS** -- reference the secret store paths for the client certificate and private key.

6. For _Security protocol_, select **Plaintext** or **SSL** for No Auth, and **SASL\_PLAINTEXT** or **SASL\_SSL** for Basic and SCRAM authentication.

7. To crawl Schema Registry subjects alongside Kafka, set _Include Schema Registry_ to **True** and provide the following details:
 - For _Schema registry host_, enter the URL of your Schema Registry endpoint (for example, `https://psrc-xxxxx.us-east-2.aws.confluent.cloud`).
 - For _API Key_, reference the secret store path where the [Schema Registry API key](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt) is stored.
 - For _API Secret_, reference the secret store path where the [Schema Registry API secret](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt) is stored.
8. Store sensitive credential values in your secret store and reference them in the corresponding fields. For more information, see [Configure secrets for workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).

9. Click **Next** after completing the configuration.

### Configure connection

Set up the connection identity and access controls for your Apache Kafka source.

1. Provide a _Connection Name_ that represents your source environment -- for example, `production`, `development`, `gold`, or `analytics`.

2. Under _Connection Admins_, add the users or groups that can manage this connection. If you leave this empty, no one can manage the connection, including admins.

3. At the bottom of the screen, click **Next**.

### Configure crawling options

On the _Metadata_ page, you can override the defaults for any of these options. If an asset appears in both include and exclude filters, the exclude filter takes precedence. When Schema Registry credentials are provided, the topic include/exclude regex also applies to schema subjects. Subjects are matched using their base topic name (stripping the `-key` or `-value` suffix).

- For _Skip internal topics_, keep the default **Yes** to skip [internal Apache Kafka topics](https://kafka.apache.org/11/documentation/streams/developer-guide/manage-topics#streams-developer-guide-topics-internal), or select **No** to crawl them.
- Click **Exclude topics regex** to exclude specific topics. Defaults to no exclusions if none are specified.
- Click **Include topics regex** to limit crawling to specific topics. Defaults to all topics if none are specified.

### Run crawler

After configuring all options, run or schedule the crawler.

1. For Direct extraction, click **Preflight checks** to validate [permissions and configuration](https://docs.atlan.com/llms/connectors/apache-kafka/preflight-checks-for-apache-kafka/llms.txt) before running. For Agent and Offline extraction, skip this step.

2. Click **Run** to run the crawler once immediately, or click **Schedule & Run** to schedule the crawler to run hourly, daily, weekly, or monthly.

Once the crawler completes, the assets appear on Atlan's asset page.

## See also

- [What does Atlan crawl from Apache Kafka](https://docs.atlan.com/llms/connectors/apache-kafka/what-does-atlan-crawl-from-apache-kafka/llms.txt): Assets and metadata discovered during crawling
- [Preflight checks for Apache Kafka](https://docs.atlan.com/llms/connectors/apache-kafka/preflight-checks-for-apache-kafka/llms.txt): Validation checks for permissions and configuration

---