Tenant logs

✦TL;DR

Learn about tenant logs.

||

|Connect

Atlan can help you understand the events that occur in your tenants, including user and administrative actions. Learn more about logging and retention as follows:

Tenant logs

Note the following:

• Load balancer logs for Azure and GCP tenants are currently not enabled.
• AuditSearch and SearchLog records are persisted forever in Elasticsearch. The 30-day retention period pertains to application logs written to logging Elasticsearch.
• An example of block storage mentioned below is Amazon Elastic Block Store (EBS) for AWS.

Production tenants

Log types	Retention	Storage	AWS	Azure	GCP
Active tenant overall backup	15 days	Object storage	✅	✅	✅
Offboarded tenant overall backup	AWS - 30 days, Azure and GCP - 15 days	Object storage	✅	✅	✅
Load balancer logs	30 days	Object storage	✅	❌	❌
Audit - user events	60 days	PostgreSQL	✅	✅	✅
Audit - admin events	Unlimited	PostgreSQL	✅	✅	✅
Application logs	30 days	Elasticsearch and object storage	✅	✅	✅
Workflow logs	90 days	ClickHouse	✅	✅	✅
Workflow artifacts	180 days	Object storage	✅	✅	✅
Application metrics	60 days	VictoriaMetrics (block storage)	✅	✅	✅

Proof of value (POV) tenants

Log types	Retention	Storage	AWS	Azure	GCP
Active tenant overall backup	15 days	Object storage	✅	✅	✅
Offboarded tenant overall backup	AWS - 3 days, Azure and GCP - 15 days	Object storage	✅	✅	✅
Load balancer logs	30 days	Object storage	✅	❌	❌
Audit - user events	60 days	PostgreSQL	✅	✅	✅
Audit - admin events	Unlimited	PostgreSQL	✅	✅	✅
Application logs	30 days	Elasticsearch and object storage	✅	✅	✅
Workflow artifacts	180 days	Object storage	✅	✅	✅
Application metrics	60 days	VictoriaMetrics (block storage)	✅	✅	✅

Atlan logs

Service	Type	Logging pipeline	Destination
Heracles	application	Fluent Bit	S3
Argo	application, server	Argo, Fluent Bit	S3
Atlas	application, audit, perf	Fluent Bit	S3
Numaflow	application	Fluent Bit	S3
Kube events	application	Fluent Bit	S3
Wisdom	application, audit	Fluent Bit	S3
Chronos	application	Fluent Bit	S3
Redis	application	Fluent Bit	S3
Kong	application, audit	Fluent Bit, PostgreSQL, Keycloak REST API	S3
Keycloak	application	Fluent Bit	S3
Elasticsearch	application	Fluent Bit	S3
Cassandra	application	Fluent Bit	S3
Heka	application	Fluent Bit	S3
Pgpool	application, server	Fluent Bit	S3
Kafka	events	Fluent Bit	S3

Cloud storage lifecycle

The cloud storage created for each tenant has its own lifecycle. The lifecycle policy is attached to paths in the cloud storage. The lifecycle policy applied to a production tenant is as follows:

Amazon Web Services (AWS)

Lifecycle policy	Path	Action
DeleteClusterLogsAfter30Days	logs/	Expires
DeleteArgoArtifactsAfter180Days	argo-artifacts/	Transitions to S3 Glacier Flexible Retrieval, then expires
DeleteArgoBackupAfter15Days	backup/argo/	Expires
DeleteAltanScheduleQuery	argo-artifacts/default/schedule-query/	Expires
DeletePostgresBackupAfter15Days	backup/postgres/	Expires
DeleteRedisBackupAfter15Days	backup/redis/	Expires
DeleteCassandraBackupAfter15Days	backup/cassandra/	Expires
DeletePrometheusBackupAfter15Days	backup/prometheus/	Expires
DeleteALBLogsAfter30Days	AWSLogs/	Expires

Microsoft Azure

Lifecycle policy	Path	Action
DeleteClusterLogsAfter30Days	logs/	Delete
DeleteArgoArtifactsAfter180Days	argo-artifacts/	Moves to archive after 90 days and delete after 180 days
DeleteArgoBackupAfter15Days	backup/argo/	Delete
DeletePostgresBackupAfter15Days	backup/postgres/	Delete
DeleteRedisBackupAfter15Days	backup/redis/	Delete
DeleteCassandraBackupAfter15Days	backup/cassandra/	Delete
DeleteAltanScheduleQuery	argo-artifacts/default/schedule-query/	Delete if blobs not modified in 1 day
DeleteSparkEventLogsAfter15Days	spark-event-logs/	Delete

Google Cloud Platform (GCP)

Lifecycle policy	Path	Action
DeleteClusterLogsAfter15Days	logs/	Delete
DeleteArgoArtifactsAfter180Days	argo-artifacts/	Archive
DeleteArgoArtifactsAfter270Days	argo-artifacts/	Delete
DeleteArgoBackupAfter3Days	backup/argo/	Delete
DeletePostgresBackupAfter3Days	backup/postgres/	Delete
DeletePrometheusBackupAfter3Days	backup/prometheus/	Delete
DeleteRedisBackupAfter3Days	backup/redis/	Delete
DeleteScheduleQueryAfter1Day	argo-artifacts/default/schedule-query/	Delete
DeleteSparkEventLogsAfter15Days	spark-event-logs/	Delete
DeleteCassandraBackupAfter3Days	backup/cassandra/	Delete