Skip to main content
Community Hub

Set up Confluent Kafka

TL;DR

Atlan supports the API authentication method for fetching metadata from Confluent Kafka. This method uses an API key and API secret to fetch metadata.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect
Who can do this?

You need your OrganizationAdmin, EnvironmentAdmin, or CloudClusterAdmin role to complete these steps. You may not have access yourself.

Atlan supports the API authentication method for fetching metadata from Confluent Kafka. This method uses an API key and API secret to fetch metadata.

If you want Atlan to also crawl schema metadata, you can connect a Confluent Schema Registry as part of the same Confluent Kafka connection. See Set up Schema Registry below for details.

Create API key​

This section provides steps for creating an API key to access metadata from your Confluent Kafka environment.

info

Atlan doesn't perform any API requests or queries that modify the resources in your Confluent Kafka environment.

Resource-specific API key​

To create a resource-specific API key for crawling Confluent Kafka, follow these steps:

  1. Log in to your Confluent Cloud instance with a OrganizationAdmin, EnvironmentAdmin, or CloudClusterAdmin role.
  2. From the Cloud Console, select your active cluster.
  3. Under Cluster Overview in the left menu for your active cluster, click API Keys.
  4. In the upper right of the API Keys page, click + Add key.
  5. On the Create key page, enter the following details:
    1. For Access Control, under Select Scope for API key, select Granular access to define a specific set of access rules, then click Next to proceed.
    2. For Service Account, click Create a new one and enter the following details:
      1. For New service account name, enter a meaningful name for your API key. For example, Atlan.
      2. For Description, add a description to identify the key later. For example, Atlan crawler connection.
      3. Click Next to proceed.
    3. For Add ACLs to service account, click + Add ACLs to add the following minimum permissions required for your Confluent Kafka resources:
      • Cluster: DescribeConfigs
      • Group: Describe
      • Topic: Describe, DescribeConfigs
    4. Once you've added all the permissions, click Next to proceed.
    5. For Create key, under Get your API key, copy or download the API key and secret. Make sure to store them securely, as the secret can't be retrieved later.

Cloud API key​

A Cloud API key lets Atlan call the Confluent Cloud APIs. Create one if you want Atlan to collect topic sizes (sizeInBytes) or build lineage from your Kafka Connect connectors. Atlan uses the same key for both. The resource-specific API key from the previous section doesn't work with the Confluent Cloud APIs, so create a separate key.

  1. Click the hamburger menu (☰) icon in the top right corner to open the menu.
  2. In the menu, click API Keys.
  3. On the API Key listing screen, click Add API Key.
  4. On the Select an account for API key screen, select the service account or user account for the key, then click Next.
  5. On the Select resource scope for API key screen, choose Cloud resource management, then click Next.
  6. On the API details screen, enter the following details:
    • Name: Enter a name that identifies the key. For example, Atlan Cloud API key.
    • Description: Describe the purpose of the key. For example, Atlan topic sizes and Connect lineage.
  7. Click Create API Key to generate the key.
  8. On the API key download screen, copy the API key and secret, or click Download API Key to save them as a file. Store them securely, as the secret can't be retrieved later.
  9. Click Complete. The key appears on the API key listing screen.
  10. Assign roles to the account that owns the key, based on the features you plan to turn on:
    • For topic sizes, assign the MetricsViewer role for your cluster.
    • For Connect lineage, assign the DeveloperRead role for the connectors in your cluster. This role lets Atlan read connector configurations. Atlan doesn't change or restart your connectors.
  11. From the Cloud Console, select your cluster, open Cluster settings, and copy the cluster ID. The ID starts with lkc-, for example lkc-xxxxx. You don't need the environment ID: Atlan finds the environment from the cluster ID.

Set up schema registry​

To crawl schema subjects and schema versions alongside Kafka topics, create a Schema Registry API key. This uses the same API key and API secret authentication method as the standalone Confluent Schema Registry connector.

Get schema registry endpoint​

  1. Log in to your Confluent Cloud instance.
  2. From the left menu of the Cloud Console, click Environments and select your environment.
  3. On your environment page, under Stream Governance API in the right menu, for Endpoint, click the clipboard icon to copy the endpoint URL (for example, https://psrc-xxxxx.us-east-2.aws.confluent.cloud) and store it in a secure location.

Create Schema Registry API key​

  1. Under Credentials in the right menu, click View & manage if there are existing API keys, or click 0 keys if there are none, to open the API credentials dialog.
  2. In the API credentials dialog, click + Add key (or Create key if there are no existing keys) to create a new Schema Registry API key.
  3. From the Create a new Schema Registry API key dialog:
    1. For Key, click the clipboard icon to copy the API key and store it in a secure location.
    2. For Secret, click the clipboard icon to copy the API secret and store it in a secure location.
    3. For Description, enter a description -- for example, Atlan Schema Registry integration.

You need the Schema Registry endpoint, API key, and API secret when configuring the crawler.

Next steps​

  • Crawl Confluent Kafka: Configure and run the crawler in Atlan, including the Cloud API and Schema Registry details you created