## What does Atlan crawl from Confluent Kafka?

URL: https://docs.atlan.com/apps/connectors/messaging/confluent-kafka/references/what-does-atlan-crawl-from-confluent-kafka

> During a Confluent Kafka crawl, Atlan extracts metadata for Clusters, Topics, Consumer Groups, Schema Subjects, Schema Versions, and Schema Fields. Reference tables map each Confluent Kafka property to its Atlan asset.

Atlan extracts metadata for Clusters, Topics, Consumer Groups, Schema Subjects, Schema Versions, and Schema Fields from Confluent Kafka and maps each property to its internal asset model. The tables below list which Confluent Kafka properties are read and how they appear on the corresponding Atlan asset. Once you've [crawled Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt), you can [use connector-specific filters](https://docs.atlan.com/llms/catalog/discovery/use-the-filters-menu/llms.txt) for quick asset discovery.

## Cluster

Atlan maps the Kafka cluster to its `KafkaCluster` asset type. One cluster asset is created per connection. You can filter clusters by bootstrap servers, cluster type, and authentication type.

| Source property | Atlan property |
| --- | --- |
| Cluster name | name |
| Bootstrap server addresses | bootstrapServers |
| Total topic count | topicCount |
| Total partition count | partitionCount |
| Total broker count | brokerCount |
| Distribution type | clusterType |
| Authentication mechanism | authenticationType |
| Schema Registry URL (if configured) | schemaRegistryUrl |

## Topics

Atlan maps topics from Confluent Kafka to its `KafkaTopic` asset type. You can filter topics by message count, size (MB), partition count, and cleanup policy.

| Source property | Atlan property |
| --- | --- |
| Topic | name |
| PartitionCount | kafkaTopicPartitionsCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| segment.bytes | kafkaTopicSegmentBytes |
| compression.type | kafkaTopicCompressionType |
| cleanup.policy | kafkaTopicLogCleanupPolicy |
| isInternal | kafkaTopicIsInternal |
| sizeInBytes | kafkaTopicSizeInBytes |
| recordCount | kafkaTopicRecordCount |
| retention.ms | kafkaTopicRetentionTimeInMs |
| retention.bytes | kafkaTopicRetentionBytes |
| Schema Registry subject name (if Schema Registry configured) | topicSchemaRegistrySubjectName |
| Whether topic is governed by a schema registry | topicIsSchemaManaged |
| Number of active consumer groups | topicConsumerCount |
| Parent cluster qualified name | topicClusterQualifiedName |

:::warning

Retrieving `sizeInBytes` requires Confluent Kafka to be set up with a [Cloud API key and secret](https://docs.atlan.com/llms/connectors/confluent-kafka/set-up-confluent-kafka/llms.txt). If the Cloud API key isn't configured, `sizeInBytes` be set to `0`.

:::

## Consumer groups

Atlan maps consumer groups from Confluent Kafka to its `KafkaConsumerGroup` asset type. You can filter consumer groups by member count and topic name. Consumer groups appear only when they're active at the time of crawling. If a topic isn't being consumed actively, Confluent Kafka purges the consumer group and it won't be cataloged as an asset.

| Source property | Atlan property |
| --- | --- |
| GROUP | name |
| memberCount | kafkaConsumerGroupMemberCount |
| topic\_names | kafkaTopicNames |
| TOPIC | kafkaConsumerGroupTopicConsumptionProperties.topicName |
| PARTITION | kafkaConsumerGroupTopicConsumptionProperties.topicPartition |
| LAG | kafkaConsumerGroupTopicConsumptionProperties.topicLag |
| CURRENT-OFFSET | kafkaConsumerGroupTopicConsumptionProperties.topicCurrentOffset |
| Consumer group state | consumerGroupState |
| Assigned topic-partition pairs | consumerGroupAssignedPartitions |

The `consumerGroupState` property reflects the current state of the consumer group at the time of crawling. Possible values are:

| Value | Description |
| --- | --- |
| `Stable` | The group has active members with all partitions assigned. |
| `Empty` | The group exists but has no active members. |
| `PreparingRebalance` | The group is collecting member join requests ahead of a rebalance. |
| `CompletingRebalance` | The group is waiting for members to receive their partition assignments. |
| `Dead` | The group has no members and its metadata has been removed. |

## Schema subjects

Atlan maps subjects from a connected Schema Registry to its `SchemaRegistrySubject` asset type. Schema subjects are crawled only when [Schema Registry credentials are provided](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt). You can filter schema subjects by schema type and compatibility.

| Source property | Atlan property |
| --- | --- |
| subject | name |
| schemaType | schemaRegistrySchemaType |
| id | schemaRegistrySchemaId |
| subject (base name without -key/-value suffix) | schemaRegistrySubjectBaseName |
| subject ends with `-key` | schemaRegistrySubjectIsKeySchema |
| compatibilityLevel | schemaRegistrySubjectSchemaCompatibility |
| version (latest) | schemaRegistrySubjectLatestSchemaVersion |
| schema (latest definition) | schemaRegistrySubjectLatestSchemaDefinition |

## Schema versions

Atlan maps each schema version to its `SchemaRegistryVersion` asset type, allowing you to track the full version history of a subject. Schema versions are crawled only when [Schema Registry credentials are provided](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt). You can filter schema versions by version number and schema type.

| Source property | Atlan property |
| --- | --- |
| version number | versionNumber |
| schema definition | versionSchemaDefinition |
| schema type (AVRO, JSON, PROTOBUF) | versionSchemaType |
| parent subject qualified name | subjectQualifiedName |

## Schema fields

Atlan maps fields from schema definitions to its `KafkaField` asset type, enabling field-level lineage between Kafka topics and downstream assets. Schema fields are crawled only when [Schema Registry credentials are provided](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt). You can filter schema fields by data type and whether they're optional or required.

| Source property | Atlan property |
| --- | --- |
| field name | name |
| field data type | fieldDataType |
| field is optional | fieldIsOptional |
| default value | fieldDefaultValue |
| schema version where field was introduced | fieldVersionIntroduced |
| field position in schema (0-based) | fieldOrder |
| parent topic qualified name | topicQualifiedName |
| schema type (key or value schema) | fieldSchemaType |

---