What does Atlan crawl from Microsoft Azure Event Hubs?
Atlan crawls and maps the following assets and properties from Microsoft Azure Event Hubs.
Atlan crawls and maps the following assets and properties from Microsoft Azure Event Hubs.
Once you've crawled Microsoft Azure Event Hubs, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:
- Event hubs - Message count, size (MB), partition count, and cleanup policy filters
- Consumer groups - Member count and topic name filters
Event hubs
Atlan maps event hubs from Microsoft Azure Event Hubs to its AzureEventHub asset type.
| Source property | Atlan property |
|---|---|
| Topic | name |
| PartitionCount | kafkaTopicPartitionsCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| segment.byte | kafkaTopicSegmentBytes |
| compression.type | kafkaTopicCompressionType |
| cleanup.policy | kafkaLogTopicCleanupPolicy |
| isInternal | kafkaTopicIsInternal |
| sizeInBytes | kafkaTopicSizeInBytes |
| recordCount | kafkaTopicRecordCount |
| status | azureEventHubStatus |
| retention.ms | kafkaTopicRetentionTimeInMs |
Consumer groups
Atlan maps consumer groups from Microsoft Azure Event Hubs to either of the following asset types:
KafkaConsumerGroup- managed via Kafka clients.AzureEventHubConsumerGroup- managed via Azure portal, SDK, or Azure Resource Manager templates.
Consumer groups are most likely to show up only in streaming scenarios. This is because if a topic is not being consumed actively, Microsoft Azure Event Hubs will purge the consumer group. So, if a consumer group is inactive while the workflow runs in Atlan, it will not be cataloged as an asset.
| Source property | Atlan property |
|---|---|
| GROUP | name |
| memberCount | kafkaConsumerGroupMemberCount |
| ReplicationFactor | kafkaTopicReplicationFactor |
| topic_names | kafkaTopicNames |
| TOPIC | kafkaConsumerGroupTopicConsumptionProperties.topicName |
| PARTITION | kafkaConsumerGroupTopicConsumptionProperties.topicPartition |
| LAG | kafkaConsumerGroupTopicConsumptionProperties.topicLag |
| CURRENT-OFFSET | kafkaConsumerGroupTopicConsumptionProperties.topicCurrentOffset |