## Crawl Aiven Kafka

URL: https://docs.atlan.com/apps/connectors/messaging/aiven-kafka/how-tos/crawl-aiven-kafka

> Extract metadata from Aiven Kafka to catalog topics, partitions, and consumer groups. Discover and govern your Kafka messaging infrastructure after configuring permissions.

Extract metadata from your [Aiven Kafka](https://docs.atlan.com/llms/connectors/aiven-kafka/set-up-aiven-kafka/llms.txt) cluster into Atlan to catalog topics, partitions, and consumer groups. After configuring the necessary permissions, you can establish a connection and crawl to extract asset-level details.

:::warning Offline extraction sunset
The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt).
:::

To crawl metadata from Aiven Kafka, review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) and then complete the following steps.

## Select source

To select Aiven Kafka as your source:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Aiven Kafka Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

## Configure connection

To complete the Aiven Kafka connection configuration:

1. Provide a _Connection Name_ that represents your source environment. For example, you might use values like `production`,`development`,`gold`, or `analytics`.

2. (Optional) To change the users who are able to manage this connection, change the users or groups listed under _Connection Admins_.

 :::warning

 If you don't specify any user or group, no one can manage the connection - not even admins.

 :::

3. Navigate to the bottom of the screen and click **Next** to proceed.

## Configure crawler

Before running the Aiven Kafka crawler, you can further configure it.

On the _Metadata_ page, you can override the defaults for any of these options:

- To select the assets you want to exclude from crawling, click **Exclude topics regex**. (This defaults to no assets, if none specified.)
- To select the assets you want to include in crawling, click **Include topics regex**. (This defaults to all assets, if none are specified.)

:::info **Did you know?** 
 If an asset appears in both the include and exclude filters, the exclude filter takes precedence.

:::

## Run crawler

To run the Aiven Kafka crawler, after completing the previous steps:

1. To check for any [permissions or other configuration issues](https://docs.atlan.com/llms/connectors/aiven-kafka/preflight-checks-for-aiven-kafka/llms.txt) before running the crawler, click **Preflight checks**.

2. You can either:
 - To run the crawler once immediately, at the bottom of the screen, click the **Run** button.
 - To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule & Run** button.

Once the crawler has completed running, the assets appear on Atlan's asset page! 🎉

---