Skip to main content
Community Hub

Preflight checks for Confluent Kafka

TL;DR

Before running the Confluent Kafka crawler, run preflight checks to perform necessary technical validations.

Your AI can read this via Docs MCPcurl -fsSL "https://docs.atlan.com/install-docs-mcp" | bashConnect

Before running the Confluent Kafka crawler, run preflight checks to perform the necessary technical validations. Checks run in order, and if a required check fails, the remaining checks don't run. Schema Registry and Cloud API checks run only when you turn on the related options. Advisory checks run last and report problems without failing the preflight result.

Kafka checks​

Authentication​

  • ✅ Check successful if Atlan authenticates with the API key and secret and can list topics.
  • ❌ Check failed if the cluster rejects the API key and secret.

When Include Schema Registry is True, this check also lists Schema Registry subjects and fails if the Schema Registry rejects its API key and secret. When Enable Cloud API is True, this check also verifies the Cloud API key. It fails if Confluent Cloud rejects the key, if Cloud API Key and Cloud API Secret hold the same value, or if Cloud API Key holds the cluster's Kafka API key instead of a Cloud API key.

Connection​

  • ✅ Check successful if Atlan connects to the cluster and lists its topics.
  • ❌ Check failed if the cluster can't be reached.

Broker reachability​

  • ✅ Check successful if every broker that the cluster reports is reachable.
  • ❌ Check failed if one or more brokers aren't reachable. The message lists the unreachable brokers. Check your network or Private Link routing.

Topic metadata​

  • ✅ Check successful if Atlan can read topic configurations and offsets.
  • ❌ Check failed if Atlan can't read topic configurations or offsets. Topics without the DescribeConfigs permission are excluded from cataloging. To catalog them, grant the API key DescribeConfigs on topics.

This check is advisory when the API key lacks DescribeConfigs on topics or when offsets can't be read. A connection error or timeout while reading topic configurations fails the preflight result.

Consumer groups​

  • ✅ Check successful if Atlan can list consumer groups.
  • ❌ Check failed if Atlan can't list consumer groups. Grant the API key Describe on consumer groups.

Consumer group metadata​

This check is advisory.

  • ✅ Check successful if Atlan can read consumer group details and offsets.
  • ❌ Check failed if Atlan can't read consumer group details or offsets.

Schema Registry checks​

These checks run when Include Schema Registry is True in the crawler credentials.

Schema Registry authentication​

  • ✅ Check successful if Atlan authenticates with the Schema Registry API key and secret.
  • ❌ Check failed if the Schema Registry rejects the API key and secret.

Schema Registry subjects​

  • ✅ Check successful if the provided credentials can list subjects in the Schema Registry.
  • ❌ Check failed if the credentials can't list subjects. Grant at least read access to the subjects.

Schema Registry schemas​

  • ✅ Check successful if the provided credentials can retrieve schema definitions from the Schema Registry.
  • ❌ Check failed if the credentials can't retrieve schema definitions.

Cloud API checks​

These checks run when Enable Cloud API is True in the crawler credentials. The Cloud API key needs the roles described in the setup guide.

Connect API​

This check runs when Include Connect Lineage is True.

  • ✅ Check successful if Atlan can list the connectors on the cluster and read their configurations. The message reports how many connector configurations are readable, or that the cluster has no connectors.
  • ❌ Check failed in the following cases:
    • The credential has no usable Cloud API configuration. Set Enable Cloud API to True and fill in Cloud API Key, Cloud API Secret, and Cluster ID, or turn off Include Connect Lineage.
    • Confluent Cloud rejects the Cloud API key and secret. Use a Cloud API key, not a Kafka API key.
    • The key's role can't list connectors or read their configurations. Grant the key the DeveloperRead role.
    • The cluster isn't found in any environment that the key can see. Check the Cluster ID, and grant the key the DeveloperRead role on that cluster.

Metrics API​

This check is advisory and runs when Include Cloud Metrics is True.

  • ✅ Check successful if Atlan can query topic sizes from the Confluent Cloud Metrics API.
  • ❌ Check failed if Atlan can't query the Metrics API, for example because the key lacks the MetricsViewer role on the cluster. Grant the role, or turn off Include Cloud Metrics. Topic sizes aren't populated while the Metrics API can't be queried.

If Include Cloud Metrics is True but the credential has no usable Cloud API configuration, the check fails and isn't advisory. Set Enable Cloud API to True and fill in the Cloud API details, or turn off the option.

Cloud API usage​

This check is advisory and always passes. It runs when Enable Cloud API is True but both Include Cloud Metrics and Include Connect Lineage are off, and reports that the run doesn't use the Cloud API key.