What does Atlan crawl from Amazon MSK?
During an Amazon MSK crawl, Atlan extracts metadata for the cluster, topics, and consumer groups. Reference tables map each Amazon MSK property to its Atlan asset.
Once you've crawled Amazon MSK, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:
- Topics - Message count, size (MB), partition count, and cleanup policy filters
- Consumer groups - Member count and topic name filters
Atlan extracts metadata for the cluster, topics, and consumer groups from Amazon MSK and maps each property to its internal asset model. The tables below list which Amazon MSK properties are read and how they appear on the corresponding Atlan asset.
Atlan creates asset-level lineage from the cluster to each topic, and from each topic to the consumer groups that read it. Atlan doesn't create column-level lineage, or lineage to assets outside Amazon MSK.
Cluster
Atlan maps the Amazon MSK cluster to its KafkaCluster asset type. One cluster asset is created per connection.
| Source property | Atlan property |
|---|---|
| Cluster name | name |
| Bootstrap server addresses | kafkaBootstrapServers |
| Total topic count | kafkaTopicCount |
| Total partition count | kafkaPartitionCount |
| Total broker count | kafkaBrokerCount |
| Authentication mechanism | kafkaAuthenticationType |
Topics
Atlan maps topics from Amazon MSK to its KafkaTopic asset type.
| Source property | Atlan property |
|---|---|
| Topic | name |
| PartitionCount | kafkaTopicPartitionsCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| segment.byte | kafkaTopicSegmentBytes |
| compression.type | kafkaTopicCompressionType |
| cleanup.policy | kafkaTopicLogCleanupPolicy |
| isInternal | kafkaTopicIsInternal |
| sizeInBytes | kafkaTopicSizeInBytes |
| recordCount | kafkaTopicRecordCount |
| retention.ms | kafkaTopicRetentionTimeInMs |
Consumer groups
Atlan maps consumer groups from Amazon MSK to its KafkaConsumerGroup asset type.
Consumer groups are most likely to show up only in streaming scenarios. This is because if a topic is not being consumed actively, Amazon MSK will purge the consumer group. So, if a consumer group is inactive while the workflow runs in Atlan, it will not be cataloged as an asset.
| Source property | Atlan property |
|---|---|
| GROUP | name |
| memberCount | kafkaConsumerGroupMemberCount |
| topic_names | kafkaTopicNames |
| TOPIC | kafkaConsumerGroupTopicConsumptionProperties.topicName |
| PARTITION | kafkaConsumerGroupTopicConsumptionProperties.topicPartition |
| LAG | kafkaConsumerGroupTopicConsumptionProperties.topicLag |
| CURRENT-OFFSET | kafkaConsumerGroupTopicConsumptionProperties.topicCurrentOffset |