Architecture

Self-Deployed Runtime enables secure metadata extraction from enterprise data sources and transfers this metadata to Atlan SaaS for processing and asset management. The runtime operates entirely within your infrastructure while maintaining secure communication with Atlan's cloud services.

Core components

Self-Deployed Runtime consists of four core components that work together to provide metadata extraction and transfer capabilities:

• Temporal worker: Orchestrates extraction workflows through persistent connections
• Dapr: Makes it easier to connect to back-end services (such as object storage and secret stores) using standard, secure APIs
• FastAPI server: Provides operational APIs for monitoring and control
• Application code: Implements source-specific extraction logic

Temporal worker

Purpose: Orchestrates metadata extraction workflows through persistent gRPC connections with Atlan's Temporal service.

Temporal workers are long-running processes that orchestrate the entire metadata extraction workflow. These workers establish persistent gRPC connections with the Temporal service hosted in Atlan's environment and continuously listen on designated task queues.

Key responsibilities:

• Poll the task queue for available extraction tasks
• Execute the extraction logic for each task
• Send results back to the Temporal service hosted by Atlan

Technical details:

• Connection type: Persistent gRPC
• Authentication: OAuth 2.0 client credentials flow with unique Client ID and Client Secret per application
• Communication pattern: Worker-initiated (no inbound connections required)
• Security model: All connections originate from your infrastructure to Atlan services, ensuring no inbound network access is required

For security details, see Security.

Dapr components

Purpose: Provides abstraction layer for secure interaction with back-end services including secret stores, object storage, and Atlan SaaS.

Dapr (Distributed Application Runtime) serves as the abstraction layer between the application and various back-end services. It provides a consistent interface for common distributed application needs, simplifying integration with different infrastructure components.

Core capabilities:

• Retrieve secrets from secret stores
• Write data to object stores
• Send events and extracted metadata to Atlan SaaS over HTTPS

Supported storage targets:

• AWS S3
• Azure Blob Storage
• Google Cloud Storage (GCS)
• Local file systems

Dapr handles all the complexity of different storage APIs and authentication mechanisms, providing a unified interface for the application code.

Learn more: Dapr

FastAPI server

Purpose: Provides REST APIs for operational control, monitoring, and observability of Self-Deployed Runtime. Each Self-Deployed Runtime application hosts its own FastAPI server that exposes REST APIs for operational management.

Available endpoints:

Endpoint	Purpose	Returns
Health check	System health monitoring	Platform, hostname, IP address, MAC address, processor, RAM
Readiness	Service availability	Confirmation if system is ready to handle requests
Observability	Operational insights	Metrics, logs, and runtime status

These APIs enable integration with existing enterprise monitoring and alerting systems.

Application code for extraction

Purpose: Implements source-specific application logic (for example metadata extraction and transformation). The application code implements the actual business logic for metadata extraction from various source systems.

Process flow: Metadata Extraction

1. Workflow trigerred by user or on schedule
2. Connect to source system based on configured secret store
3. Fetch metadata based on parameters
4. Transform metadata into standardized formats (JSON or Parquet)
5. Write transformed data to an object store
6. Transfer metadata to Atlan SaaS for further processing

Data flow

Self-Deployed Runtime implements a controlled pipeline for metadata extraction and transfer:

Source Systems → Extraction → Local Storage → Object Store → Atlan SaaS
                     ↓             ↓              ↓             ↓
                Application    Temporary      Persistent      Cloud
                   Code          Files         Storage       Processing

How it works

The complete workflow follows these steps:

1. Deploy and register: Applications are deployed on your enterprise infrastructure and registered with your Atlan tenant. This establishes the secure connection between your environment and Atlan services using OAuth 2.0 client credentials.

2. Configure workflow: A user configures a workflow from the Atlan UI, defining source systems to connect to, extraction schedules or triggers, transformation requirements, and target storage locations.

3. Execute extraction: The application retrieves relevant job details from Atlan and performs defined actions to extract metadata from the source. This includes:

   • Connecting to specified source systems using appropriate protocols
   • Executing extraction logic based on configured parameters
   • Transforming metadata into standardized formats (JSON or Parquet)

4. Transfer metadata: Extracted metadata is first written to configured storage (S3, Azure Blob, GCS, or others), and then securely transferred to the Atlan SaaS tenant using HTTPS.

5. Process and publish: Atlan workflows processes the transferred metadata files and publishes them as searchable, governed assets in your Atlan workspace.

6. Monitor execution: Throughout the execution, logs are collected and sent to Atlan for monitoring and auditing. Status updates are shown in the Atlan UI, and health metrics are exposed via FastAPI endpoints for integration with enterprise monitoring systems.

See also

• Authentication: How OAuth 2.0 authentication works and protects your infrastructure.
• Data transfer and observability: How metadata moves from sources to Atlan and monitoring capabilities.
• Deployment options: How containerization enables flexible deployment across environments.
• Security: Security architecture, authentication, encryption, and compliance controls for Self-Deployed Runtime.
• Feature differences: Compare Self-Deployed Runtime and Atlan-hosted execution modes.