What does Atlan crawl from Apache Kafka?
During an Apache Kafka crawl, Atlan extracts metadata for Clusters, Topics, Consumer Groups, Schema Subjects, Schema Versions, and Schema Fields. Reference tables map each Kafka property to its Atlan asset.
Atlan extracts metadata for Clusters, Topics, Consumer Groups, Schema Subjects, Schema Versions, and Schema Fields from Apache Kafka and maps each property to its internal asset model. The tables below list which Apache Kafka properties are read and how they appear on the corresponding Atlan asset. Once you've crawled Apache Kafka, you can use connector-specific filters for quick asset discovery.
Cluster
Atlan maps the Kafka cluster to its KafkaCluster asset type. One cluster asset is created per connection. You can filter clusters by bootstrap servers and authentication type.
| Source property | Atlan property |
|---|---|
| Cluster name | name |
| Bootstrap server addresses | kafkaBootstrapServers |
| Total topic count | kafkaTopicCount |
| Total partition count | kafkaPartitionCount |
| Total broker count | kafkaBrokerCount |
| Authentication mechanism | kafkaAuthenticationType |
| Schema Registry URL (if configured) | kafkaSchemaRegistryUrl |
Topics
Atlan maps topics from Apache Kafka to its KafkaTopic asset type. You can filter topics by message count, size (MB), partition count, and cleanup policy.
| Source property | Atlan property |
|---|---|
| Topic | name |
| PartitionCount | kafkaTopicPartitionsCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| segment.bytes | kafkaTopicSegmentBytes |
| compression.type | kafkaTopicCompressionType |
| cleanup.policy | kafkaTopicLogCleanupPolicy |
| isInternal | kafkaTopicIsInternal |
| sizeInBytes | kafkaTopicSizeInBytes |
| recordCount | kafkaTopicRecordCount |
| retention.ms | kafkaTopicRetentionTimeInMs |
| retention.bytes | kafkaTopicRetentionBytes |
| Schema Registry subject name (if Schema Registry configured) | kafkaTopicSchemaRegistrySubjectName |
| Whether topic is governed by a schema registry | kafkaTopicIsSchemaManaged |
| Number of active consumer groups | kafkaTopicConsumerCount |
| Parent cluster qualified name | kafkaTopicClusterQualifiedName |
Consumer groups
Atlan maps consumer groups from Apache Kafka to its KafkaConsumerGroup asset type. You can filter consumer groups by member count and topic name. Consumer groups appear only when they're active at the time of crawling. If a topic isn't being consumed actively, Apache Kafka purges the consumer group and it won't be cataloged as an asset.
| Source property | Atlan property |
|---|---|
| GROUP | name |
| memberCount | kafkaConsumerGroupMemberCount |
| topic_names | kafkaTopicNames |
| TOPIC | kafkaConsumerGroupTopicConsumptionProperties.topicName |
| PARTITION | kafkaConsumerGroupTopicConsumptionProperties.topicPartition |
| LAG | kafkaConsumerGroupTopicConsumptionProperties.topicLag |
| CURRENT-OFFSET | kafkaConsumerGroupTopicConsumptionProperties.topicCurrentOffset |
| Consumer group state | kafkaConsumerGroupState |
| Assigned topic-partition pairs | kafkaConsumerGroupAssignedPartitions |
The consumerGroupState property reflects the current state of the consumer group at the time of crawling. Possible values are:
| Value | Description |
|---|---|
Stable | The group has active members with all partitions assigned. |
Empty | The group exists but has no active members. |
PreparingRebalance | The group is collecting member join requests ahead of a rebalance. |
CompletingRebalance | The group is waiting for members to receive their partition assignments. |
Dead | The group has no members and its metadata has been removed. |
Schema subjects
Atlan maps subjects from a connected Schema Registry to its SchemaRegistrySubject asset type. Schema subjects are crawled only when Schema Registry credentials are provided. You can filter schema subjects by schema type and compatibility.
| Source property | Atlan property |
|---|---|
| subject | name |
| schemaType | schemaRegistrySchemaType |
| id | schemaRegistrySchemaId |
| subject (base name without -key/-value suffix) | schemaRegistrySubjectBaseName |
subject ends with -key | schemaRegistrySubjectIsKeySchema |
| compatibilityLevel | schemaRegistrySubjectSchemaCompatibility |
| version (latest) | schemaRegistrySubjectLatestSchemaVersion |
| schema (latest definition) | schemaRegistrySubjectLatestSchemaDefinition |
Schema versions
Atlan maps each schema version to its SchemaRegistryVersion asset type, allowing you to track the full version history of a subject. Schema versions are crawled only when Schema Registry credentials are provided. You can filter schema versions by version number and schema type.
| Source property | Atlan property |
|---|---|
| version number | schemaRegistryVersionNumber |
| schema definition | schemaRegistryVersionSchemaDefinition |
| schema type (AVRO, JSON, PROTOBUF) | schemaRegistryVersionSchemaType |
| parent subject qualified name | schemaRegistrySubjectQualifiedName |
Schema fields
Atlan maps fields from schema definitions to its KafkaField asset type. Atlan doesn't create field-level lineage for these fields. Schema fields are crawled only when Schema Registry credentials are provided. You can filter schema fields by data type and whether they're optional or required.
| Source property | Atlan property |
|---|---|
| field name | name |
| field data type | kafkaFieldDataType |
| field is optional | kafkaFieldIsOptional |
| default value | kafkaFieldDefaultValue |
| field position in schema (0-based) | kafkaFieldOrder |
| parent topic qualified name | kafkaTopicQualifiedName |
| schema type (key or value schema) | kafkaFieldSchemaType |