Atlan can help you understand the events that occur in your tenants, including user and administrative actions. Learn more about logging and retention as follows:
Tenant logs
Note the following:
- Load balancer logs for Azure and GCP tenants are currently not enabled.
- AuditSearch and SearchLog records are persisted forever in Elasticsearch. The 30-day retention period pertains to application logs written to logging Elasticsearch.
- An example of block storage mentioned below is Amazon Elastic Block Store (EBS) for AWS.
Production tenants
| Log types | Retention | Storage | AWS | Azure | GCP |
|---|
| Active tenant overall backup | 15 days | Object storage | ✅ | ✅ | ✅ |
| Offboarded tenant overall backup | AWS - 30 days, Azure and GCP - 15 days | Object storage | ✅ | ✅ | ✅ |
| Load balancer logs | 30 days | Object storage | ✅ | ❌ | ❌ |
| Audit - user events | 60 days | PostgreSQL | ✅ | ✅ | ✅ |
| Audit - admin events | Unlimited | PostgreSQL | ✅ | ✅ | ✅ |
| Application logs | 30 days | Elasticsearch and object storage | ✅ | ✅ | ✅ |
| Workflow logs | 90 days | ClickHouse | ✅ | ✅ | ✅ |
| Workflow artifacts | 180 days | Object storage | ✅ | ✅ | ✅ |
| Application metrics | 60 days | VictoriaMetrics (block storage) | ✅ | ✅ | ✅ |
Proof of value (POV) tenants
| Log types | Retention | Storage | AWS | Azure | GCP |
|---|
| Active tenant overall backup | 15 days | Object storage | ✅ | ✅ | ✅ |
| Offboarded tenant overall backup | AWS - 3 days, Azure and GCP - 15 days | Object storage | ✅ | ✅ | ✅ |
| Load balancer logs | 30 days | Object storage | ✅ | ❌ | ❌ |
| Audit - user events | 60 days | PostgreSQL | ✅ | ✅ | ✅ |
| Audit - admin events | Unlimited | PostgreSQL | ✅ | ✅ | ✅ |
| Application logs | 30 days | Elasticsearch and object storage | ✅ | ✅ | ✅ |
| Workflow artifacts | 180 days | Object storage | ✅ | ✅ | ✅ |
| Application metrics | 60 days | VictoriaMetrics (block storage) | ✅ | ✅ | ✅ |
Atlan logs
| Service | Type | Logging pipeline | Destination |
|---|
| Heracles | application | Fluent Bit | S3 |
| Argo | application, server | Argo, Fluent Bit | S3 |
| Atlas | application, audit, perf | Fluent Bit | S3 |
| Numaflow | application | Fluent Bit | S3 |
| Kube events | application | Fluent Bit | S3 |
| Wisdom | application, audit | Fluent Bit | S3 |
| Chronos | application | Fluent Bit | S3 |
| Redis | application | Fluent Bit | S3 |
| Kong | application, audit | Fluent Bit, PostgreSQL, Keycloak REST API | S3 |
| Keycloak | application | Fluent Bit | S3 |
| Elasticsearch | application | Fluent Bit | S3 |
| Cassandra | application | Fluent Bit | S3 |
| Heka | application | Fluent Bit | S3 |
| Pgpool | application, server | Fluent Bit | S3 |
| Kafka | events | Fluent Bit | S3 |
Cloud storage lifecycle
The cloud storage created for each tenant has its own lifecycle. The lifecycle policy is attached to paths in the cloud storage. The lifecycle policy applied to a production tenant is as follows:
Amazon Web Services (AWS)
| Lifecycle policy | Path | Action |
|---|
DeleteClusterLogsAfter30Days | logs/ | Expires |
DeleteArgoArtifactsAfter180Days | argo-artifacts/ | Transitions to S3 Glacier Flexible Retrieval, then expires |
DeleteArgoBackupAfter15Days | backup/argo/ | Expires |
DeleteAltanScheduleQuery | argo-artifacts/default/schedule-query/ | Expires |
DeletePostgresBackupAfter15Days | backup/postgres/ | Expires |
DeleteRedisBackupAfter15Days | backup/redis/ | Expires |
DeleteCassandraBackupAfter15Days | backup/cassandra/ | Expires |
DeletePrometheusBackupAfter15Days | backup/prometheus/ | Expires |
DeleteALBLogsAfter30Days | AWSLogs/ | Expires |
Microsoft Azure
| Lifecycle policy | Path | Action |
|---|
DeleteClusterLogsAfter30Days | logs/ | Delete |
DeleteArgoArtifactsAfter180Days | argo-artifacts/ | Moves to archive after 90 days and delete after 180 days |
DeleteArgoBackupAfter15Days | backup/argo/ | Delete |
DeletePostgresBackupAfter15Days | backup/postgres/ | Delete |
DeleteRedisBackupAfter15Days | backup/redis/ | Delete |
DeleteCassandraBackupAfter15Days | backup/cassandra/ | Delete |
DeleteAltanScheduleQuery | argo-artifacts/default/schedule-query/ | Delete if blobs not modified in 1 day |
DeleteSparkEventLogsAfter15Days | spark-event-logs/ | Delete |
| Lifecycle policy | Path | Action |
|---|
DeleteClusterLogsAfter15Days | logs/ | Delete |
DeleteArgoArtifactsAfter180Days | argo-artifacts/ | Archive |
DeleteArgoArtifactsAfter270Days | argo-artifacts/ | Delete |
DeleteArgoBackupAfter3Days | backup/argo/ | Delete |
DeletePostgresBackupAfter3Days | backup/postgres/ | Delete |
DeletePrometheusBackupAfter3Days | backup/prometheus/ | Delete |
DeleteRedisBackupAfter3Days | backup/redis/ | Delete |
DeleteScheduleQueryAfter1Day | argo-artifacts/default/schedule-query/ | Delete |
DeleteSparkEventLogsAfter15Days | spark-event-logs/ | Delete |
DeleteCassandraBackupAfter3Days | backup/cassandra/ | Delete |