What does Atlan crawl from Apache Kafka?
Atlan crawls and maps the following assets and properties from Apache Kafka. Once you've crawled Apache Kafka, you can use connector-specific filters for quick asset discovery.
Cluster
Atlan maps the Kafka cluster to its KafkaCluster asset type. One cluster asset is created per connection. You can filter clusters by bootstrap servers, cluster type, and authentication type.
| Source property | Atlan property |
|---|---|
| Cluster name | name |
| Bootstrap server addresses | bootstrapServers |
| Total topic count | topicCount |
| Total partition count | partitionCount |
| Total broker count | brokerCount |
| Distribution type | clusterType |
| Authentication mechanism | authenticationType |
| Schema Registry URL (if configured) | schemaRegistryUrl |
Topics
Atlan maps topics from Apache Kafka to its KafkaTopic asset type. You can filter topics by message count, size (MB), partition count, and cleanup policy.
| Source property | Atlan property |
|---|---|
| Topic | name |
| PartitionCount | kafkaTopicPartitionsCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| segment.bytes | kafkaTopicSegmentBytes |
| compression.type | kafkaTopicCompressionType |
| cleanup.policy | kafkaTopicLogCleanupPolicy |
| isInternal | kafkaTopicIsInternal |
| sizeInBytes | kafkaTopicSizeInBytes |
| recordCount | kafkaTopicRecordCount |
| retention.ms | kafkaTopicRetentionTimeInMs |
| retention.bytes | kafkaTopicRetentionBytes |
| Schema Registry subject name (if Schema Registry configured) | topicSchemaRegistrySubjectName |
| Whether topic is governed by a schema registry | topicIsSchemaManaged |
| Number of active consumer groups | topicConsumerCount |
| Parent cluster qualified name | topicClusterQualifiedName |
Consumer groups
Atlan maps consumer groups from Apache Kafka to its KafkaConsumerGroup asset type. You can filter consumer groups by member count and topic name. Consumer groups appear only when they're active at the time of crawling. If a topic isn't being consumed actively, Apache Kafka purges the consumer group and it won't be cataloged as an asset.
| Source property | Atlan property |
|---|---|
| GROUP | name |
| memberCount | kafkaConsumerGroupMemberCount |
| topic_names | kafkaTopicNames |
| TOPIC | kafkaConsumerGroupTopicConsumptionProperties.topicName |
| PARTITION | kafkaConsumerGroupTopicConsumptionProperties.topicPartition |
| LAG | kafkaConsumerGroupTopicConsumptionProperties.topicLag |
| CURRENT-OFFSET | kafkaConsumerGroupTopicConsumptionProperties.topicCurrentOffset |
| Consumer group state | consumerGroupState |
| Assigned topic-partition pairs | consumerGroupAssignedPartitions |
The consumerGroupState property reflects the current state of the consumer group at the time of crawling. Possible values are:
| Value | Description |
|---|---|
Stable | The group has active members with all partitions assigned. |
Empty | The group exists but has no active members. |
PreparingRebalance | The group is collecting member join requests ahead of a rebalance. |
CompletingRebalance | The group is waiting for members to receive their partition assignments. |
Dead | The group has no members and its metadata has been removed. |
Schema subjects
Atlan maps subjects from a connected Schema Registry to its SchemaRegistrySubject asset type. Schema subjects are crawled only when Schema Registry credentials are provided. You can filter schema subjects by schema type and compatibility.
| Source property | Atlan property |
|---|---|
| subject | name |
| schemaType | schemaRegistrySchemaType |
| id | schemaRegistrySchemaId |
| subject (base name without -key/-value suffix) | schemaRegistrySubjectBaseName |
subject ends with -key | schemaRegistrySubjectIsKeySchema |
| compatibilityLevel | schemaRegistrySubjectSchemaCompatibility |
| version (latest) | schemaRegistrySubjectLatestSchemaVersion |
| schema (latest definition) | schemaRegistrySubjectLatestSchemaDefinition |
Schema versions
Atlan maps each schema version to its SchemaRegistryVersion asset type, allowing you to track the full version history of a subject. Schema versions are crawled only when Schema Registry credentials are provided. You can filter schema versions by version number and schema type.
| Source property | Atlan property |
|---|---|
| version number | versionNumber |
| schema definition | versionSchemaDefinition |
| schema type (AVRO, JSON, PROTOBUF) | versionSchemaType |
| parent subject qualified name | subjectQualifiedName |
Schema fields
Atlan maps fields from schema definitions to its KafkaField asset type, enabling field-level lineage between Kafka topics and downstream assets. Schema fields are crawled only when Schema Registry credentials are provided. You can filter schema fields by data type and whether they're optional or required.
| Source property | Atlan property |
|---|---|
| field name | name |
| field data type | fieldDataType |
| field is optional | fieldIsOptional |
| default value | fieldDefaultValue |
| schema version where field was introduced | fieldVersionIntroduced |
| field position in schema (0-based) | fieldOrder |
| parent topic qualified name | topicQualifiedName |
| schema type (key or value schema) | fieldSchemaType |