What does Atlan crawl from Confluent Kafka?
Atlan crawls and maps the following assets and properties from Confluent Kafka.
Once you've crawled Confluent Kafka, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:
- Topics - Message count, size (MB), partition count, and cleanup policy filters
- Consumer groups - Member count and topic name filters
Topics
Atlan maps topics from Confluent Kafka to its KafkaTopic asset type.
| Source property | Atlan property |
|---|---|
| Topic | name |
| PartitionCount | kafkaTopicPartitionsCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| segment.byte | kafkaTopicSegmentBytes |
| compression.type | kafkaTopicCompressionType |
| cleanup.policy | kafkaLogTopicCleanupPolicy |
| isInternal | kafkaTopicIsInternal |
| sizeInBytes | kafkaTopicSizeInBytes |
| recordCount | kafkaTopicRecordCount |
| retention.ms | kafkaTopicRetentionTimeInMs |
Retrieving sizeInBytes Confluent Kafka to be set up with a Cloud API key and secret. If the Cloud API key is not configured or the API key and Secret are not set, sizeInBytes will be set to 0.
Consumer groups
Atlan maps consumer groups from Confluent Kafka to its KafkaConsumerGroup asset type.
Consumer groups are most likely to show up only in streaming scenarios. This is because if a topic is not being consumed actively, Confluent Kafka will purge the consumer group. So, if a consumer group is inactive while the workflow runs in Atlan, it will not be cataloged as an asset.
| Source property | Atlan property |
|---|---|
| GROUP | name |
| memberCount | kafkaConsumerGroupMemberCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| topic_names | kafkaTopicNames |
| TOPIC | kafkaConsumerGroupTopicConsumptionProperties.topicName |
| PARTITION | kafkaConsumerGroupTopicConsumptionProperties.topicPartition |
| LAG | kafkaConsumerGroupTopicConsumptionProperties.topicLag |
| CURRENT-OFFSET | kafkaConsumerGroupTopicConsumptionProperties.topicCurrentOffset |