## Crawl Confluent Kafka

URL: https://docs.atlan.com/apps/connectors/messaging/confluent-kafka/how-tos/crawl-confluent-kafka

> Learn about crawl confluent kafka.

Atlan crawls metadata from your Confluent Kafka cluster, allowing you to discover, classify, and govern your Kafka topics and schemas. This guide walks you through the steps to configure and run the Confluent Kafka crawler in Atlan.

## Prerequisites

Before you begin, complete the following prerequisites:

- **Confluent Kafka setup:** You have [configured the Confluent Kafka permissions](https://docs.atlan.com/llms/connectors/confluent-kafka/set-up-confluent-kafka/llms.txt), you can establish a connection between Atlan and Confluent Kafka.
- **Order of operations:** Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) to understand the sequence of tasks for crawling metadata.
- **Access to Atlan workspace:** You must have the required permissions in Atlan to create and manage a connection.

## Select source

To select Confluent Kafka as your source:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Confluent Kafka Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

## Provide credentials

In **Direct extraction**, Atlan connects to Confluent Kafka and crawls metadata directly.

:::warning Offline extraction sunset
The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt).
:::

### Direct extraction method

To enter your Confluent Kafka credentials:

1. For _Bootstrap servers_, enter the hostnames of your Confluent Kafka brokers. Separate multiple hostnames with a comma `,` or semicolon `;`.

2. For _API Key_, enter the API key you copied.

3. For _API Secret_, enter the API secret you copied.

4. For Security protocol, click **SASL\_PLAINTEXT** to connect to Confluent Kafka through a non-encrypted channel or click **SASL\_SSL** to connect via a Secure Sockets Layer (SSL) channel.

5. Click the **Test Authentication** button to confirm connectivity to Confluent Kafka.

6. Once authentication is successful, navigate to the bottom of the screen and click **Next**.

## Configure connection

To complete the Confluent Kafka connection configuration:

1. Provide a _Connection Name_ that represents your source environment. For example, use values like `production`, `development`, `gold`, or `analytics`.

2. (Optional) To change the users who can manage this connection, update the users or groups listed under _Connection Admins_.

 :::warning

 If you don't specify any user or group, no one can manage the connection - not even admins.

 :::

3. Navigate to the bottom of the screen and click **Next** to proceed.

## Configure crawler

Before running the Confluent Kafka crawler, you can further configure it.

On the _Metadata_ page, you can override the defaults for any of these options:

- For _Skip internal topics_, keep the default option **Yes** to skip [internal Kafka topics](https://kafka.apache.org/11/documentation/streams/developer-guide/manage-topics#streams-developer-guide-topics-internal) or click **No** to enable crawling them.
- To select the assets you want to exclude from crawling, click **Exclude topics regex**. (This defaults to no assets, if none specified.)
- To select the assets you want to include in crawling, click **Include topics regex**. (This defaults to all assets, if none are specified.)

:::info **Did you know?** 
 If an asset appears in both the include and exclude filters, the exclude filter takes precedence.

:::

## Run crawler

After configuring all options, run or schedule the crawler.

1. For Direct extraction, click **Preflight checks** to validate [permissions and configuration](https://docs.atlan.com/llms/connectors/confluent-kafka/preflight-checks-for-confluent-kafka/llms.txt) before running. For Offline extraction, skip this step.

2. Click **Run** to run the crawler once immediately, or click **Schedule & Run** to schedule the crawler to run hourly, daily, weekly, or monthly.

Once the crawler completes, the assets appear on Atlan's asset page.

## See also

- [What does Atlan crawl from Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/what-does-atlan-crawl-from-confluent-kafka/llms.txt): Assets and metadata discovered during crawling
- [Preflight checks for Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/preflight-checks-for-confluent-kafka/llms.txt): Validation checks for permissions and configuration

---