Skip to main content

Data transfer and observability

Data transfer in Self-Deployed Runtime moves extracted metadata from your source systems to Atlan through a secure, multi-stage process. Understanding this flow helps you see how your data stays protected and where you get visibility into operations throughout the metadata extraction journey.

How metadata moves to Atlan​

After metadata is extracted from source systems, it follows a secure transfer process to Atlan SaaS for further processing:

  • App writes output to container local volume: The application initially stores extracted metadata in the container's local storage, which uses EBS volumes or node disks.
  • Output moves to persistent storage: The metadata is then periodically transferred to your configured persistent storage systems, such as local volume mounts or cloud object storage like AWS S3, Azure Blob Storage, or Google Cloud Storage.
  • Data replicates to Atlan SaaS tenant storage: Using OAuth-based client credentials for secure authentication, the metadata is replicated from your storage to your specific Atlan SaaS tenant's storage infrastructure.
  • Transfer happens via Dapr abstraction: The actual transfer occurs through Dapr, which handles the complexity of connecting to Atlan's data storage service within your tenant and routing data to Atlan's tenant-configured object storage.
  • Processing and persistence in Atlan: Once the metadata reaches your Atlan SaaS tenant, it gets processed by Atlan workflows and persisted in the Atlan metastore where it becomes searchable and governable.

How data stays encrypted​

  • Data transfer to Atlan: All communication with Atlan services is encrypted in transit using TLS 1.2+ by default, protecting metadata as it moves from your infrastructure to Atlan.
  • Your cloud storage: Any data written to your cloud storage systems like S3, Azure buckets, or GCS is encrypted at rest using your configured bucket encryption settings, typically AES-256.
  • Atlan storage: When data reaches Atlan's infrastructure, Atlan encrypts your metadata using AES-256 encryption in its cloud storage buckets.

How you can monitor operations​

Self-Deployed Runtime provides visibility into all its operations so you can track what's happening, troubleshoot issues, and monitor performance. You can monitor using logs, traces, and metrics that capture detailed information about extraction job executions, interactions with Atlan services, and connections to your data sources and secret stores.

You can store these logs in your own infrastructure (for example, using S3 buckets). The logs are already in OpenTelemetry Protocol (OTLP) format, which makes them compatible with popular monitoring systems and security information and event management (SIEM) tools.

See also​