## Crawl on-premises Kafka

URL: https://docs.atlan.com/apps/connectors/messaging/on-premises-event-buses/how-tos/crawl-on-premises-kafka

> Extract metadata from on-premises Kafka to catalog topics, partitions, and consumer groups. Discover and govern your Kafka messaging infrastructure after configuring extractor tool.

:::danger **Deprecated**
This job execution mode was deprecated on June 30, 2026 and is no longer supported or maintained, including bug fixes. Existing workflows may break/be disabled without warning. For all implementations, switch to [Self-deployed runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt).
:::

Extract metadata from your [on-premises Kafka](https://docs.atlan.com/llms/connectors/redpanda-kafka/set-up-on-premises-kafka-access/llms.txt) instances into Atlan to catalog topics, partitions, and consumer groups. After setting up the kafka-extractor tool, you can extract metadata by completing the following steps.

## Run Kafka extractor

### Crawl all Kafka connections

To crawl all Kafka connections using the kafka-extractor tool:

1. Log into the server with Docker Compose installed.
2. Change to the directory containing the compose file.
3. Run Docker Compose: `sudo docker-compose up`

### Crawl specific connection

To crawl a specific Kafka connection using the kafka-extractor tool:

1. Log into the server with Docker Compose installed.
2. Change to the directory containing the compose file.
3. Run Docker Compose: `sudo docker-compose up <connection-name>`

(Replace `<connection-name>` with the name of the connection from the `services` section of the compose file.)

## (Optional) Review generated files

The kafka-extractor tool generates many folders with JSON files for each `service`. For example:

- `topics`
- `topic-configs`
- `consumer-groups`
- `consumer-groups-members`
- and many others

You can inspect the metadata and make sure it's acceptable for providing metadata to Atlan.

## Upload generated files to object storage

To provide Atlan access to the extracted metadata, you need to upload the metadata to object storage.

### AWS S3

To upload the metadata to S3:

1. Make sure all files for a particular connection have the same prefix.
2. [Upload the files to the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) using your preferred method. Include all the files from the output folder generated after running Docker Compose.

For example, to upload all files using the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html):

```bash
aws s3 cp output/kafka-example s3://my-bucket/metadata/kafka-example --recursive
```

### Google Cloud Storage

To upload the metadata to GCS:

1. Make sure all files for a particular connection have the same prefix.
2. [Upload the files to the GCS bucket](https://cloud.google.com/storage/docs/uploading-objects) using your preferred method. Include all the files from the output folder generated after running Docker Compose.

For example, to upload all files using the [gcloud CLI](https://cloud.google.com/sdk/docs/install):

```bash
gcloud storage cp output/kafka-example gs://my-bucket/metadata/kafka-example --recursive
```

## Crawl metadata in Atlan

Once you have extracted metadata on-premises and uploaded the results to object storage, you can crawl the metadata into Atlan:

- [How to crawl Apache Kafka](https://docs.atlan.com/llms/connectors/apache-kafka/crawl-apache-kafka/llms.txt)
- [How to crawl Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt)
- [How to crawl Aiven Kafka](https://docs.atlan.com/llms/connectors/aiven-kafka/crawl-aiven-kafka/llms.txt)
- [How to crawl Redpanda Kafka](https://docs.atlan.com/llms/connectors/redpanda-kafka/crawl-redpanda-kafka/llms.txt)

---