## Set up Confluent Kafka

URL: https://docs.atlan.com/apps/connectors/messaging/confluent-kafka/how-tos/set-up-confluent-kafka

> Atlan supports the API authentication method for fetching metadata from Confluent Kafka. This method uses an API key and API secret to fetch metadata.

:::warning Who can do this?
 You will need your [OrganizationAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/rbac/predefined-rbac-roles.html#organizationadmin), [EnvironmentAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/rbac/predefined-rbac-roles.html#environmentadmin), or [CloudClusterAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/rbac/predefined-rbac-roles.html#cloudclusteradmin) role to complete these steps - you may not have access yourself.

:::

Atlan supports the API authentication method for fetching metadata from Confluent Kafka. This method uses an API key and API secret to fetch metadata.

If you want Atlan to also crawl schema metadata, you can connect a [Confluent Schema Registry](https://docs.atlan.com/llms/connectors/confluent-schema-registry/llms.txt) as part of the same Confluent Kafka connection. See [Set up Schema Registry](#set-up-schema-registry) below for details.

## Create API key

This section provides steps for creating an API key to access metadata from your Confluent Kafka environment.

:::info

💪 **Note:** Atlan does **not** perform any API requests or queries that modify the resources in your Confluent Kafka environment.

:::

### Resource-Specific API Key

To [create a resource-specific API key](https://docs.confluent.io/cloud/current/access-management/authenticate/api-keys/api-keys.html#create-a-resource-specific-api-key) for [crawling Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt), follow these steps:

1. Log in to your [Confluent Cloud](https://confluent.cloud/) instance with a [OrganizationAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/rbac/predefined-rbac-roles.html#organizationadmin), [EnvironmentAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/rbac/predefined-rbac-roles.html#environmentadmin), or [CloudClusterAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/rbac/predefined-rbac-roles.html#cloudclusteradmin) role.
2. From the _Cloud Console_, select your active cluster.
3. Under _Cluster Overview_ in the left menu for your active cluster, click **API Keys**.
4. In the upper right of the _API Keys_ page, click **\+ Add key**.
5. On the _Create key_ page, enter the following details:
 1. For _Access Control_, under _Select Scope for API key_, select **Granular access** to define a specific set of access rules, then click **Next** to proceed.
 2. For _Service Account_, click **Create a new one** and enter the following details:
 1. For _New service account name_, enter a meaningful name for your API key. For example, `Atlan`.
 2. (Optional) For _Description_, add a description for your API key. For example, `Atlan crawler connection`.
 3. Click **Next** to proceed.
 3. For _Add ACLs to service account_, click **\+ Add ACLs** to add and allow the following minimum permissions required for your Confluent Kafka resources: 
 - Cluster: `DescribeConfigs`
 - Group: `Describe`
 - Topic: `Describe`, `DescribeConfigs`
 4. Once you've added all the permissions, click **Next** to proceed.
 5. For _Create key_, under _Get your API key_, copy or download the API key and secret. Make sure to store them securely, as the secret can't be retrieved later.

### (Optional) Cloud API Key

To access Kafka metrics(`sizeInBytes`), you need a Cloud API key. Follow these steps to generate one:

1. Click the hamburger menu (☰) icon in the top right corner to open the menu.
2. In the menu, click **API Keys**.
3. On the API Key listing screen, click the **Add API Key** button to add a new API key.
4. On the _Select an account for API key_ screen, select the account appropriate for your service or user account, then click **Next**.
5. On the _Select resource scope for API key_ screen, choose **Cloud resource management**, then click **Next**.
6. On the _API details screen_, enter the required details:

- **Name**: Provide a unique and meaningful name for your API key. For example, Atlan Kafka Metrics Key.
- **Description**: Add a description that illustrates the purpose of the key. For example, Atlan Kafka metrics API key with read-only permissions.

8. Click **Create API Key** to generate the key.
9. On the _API key download screen_, copy the API key and secret or click **Download API Key** to save them as a file. Make sure to store them securely, as the secret can't be retrieved later.
10. Click **Complete** to finish the API key creation process. The generated API key appears on the API key listing screen.

## Set up schema registry

To crawl schema subjects and schema versions alongside Kafka topics, create a Schema Registry API key. This uses the same API key and API secret authentication method as the [standalone Confluent Schema Registry connector](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt).

### Get schema registry endpoint

1. Log in to your [Confluent Cloud](https://confluent.cloud/) instance.
2. From the left menu of the _Cloud Console_, click **Environments** and select your environment.
3. On your environment page, under _Stream Governance API_ in the right menu, for _Endpoint_, click the clipboard icon to copy the endpoint URL (for example, `https://psrc-xxxxx.us-east-2.aws.confluent.cloud`) and store it in a secure location.

### Create Schema Registry API key

1. Under _Credentials_ in the right menu, click **View & manage** if there are existing API keys, or click **0 keys** if there are none, to open the API credentials dialog.
2. In the _API credentials_ dialog, click **+ Add key** (or **Create key** if there are no existing keys) to create a new Schema Registry API key.
3. From the _Create a new Schema Registry API key_ dialog:
 1. For _Key_, click the clipboard icon to copy the API key and store it in a secure location.
 2. For _Secret_, click the clipboard icon to copy the API secret and store it in a secure location.
 3. For _Description_, enter a description -- for example, `Atlan Schema Registry integration`.

You need the Schema Registry endpoint, API key, and API secret when [configuring the crawler](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt).

## Next steps

- **Crawl Confluent Kafka**: Proceed to [Crawl Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt) to configure and run the crawler in Atlan.
- **Crawl with Schema Registry**: If you created Schema Registry credentials, enable Schema Registry crawling in the [Confluent Kafka crawl workflow](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt).

---