Skip to main content

How Atlan connects to Iceberg

Atlan connects to your Iceberg REST catalog to extract metadata including catalogs, namespaces, tables, and columns while maintaining network security and compliance. You can choose between Direct connectivity for catalogs available from the internet or Self-deployed runtime for catalogs that must remain behind your firewall.

Connect via direct network connection​

Atlan's Iceberg workflow establishes a direct network connection to your Iceberg REST catalog from the Atlan SaaS tenant. This approach works when your REST catalog can accept connections from the internet.

  • Atlan's Iceberg workflow connects directly to your REST catalog from the Atlan SaaS tenant over HTTPS.
  • You provide connection details (REST catalog URI, authentication token) when creating a crawler workflow.
  • Atlan executes API calls to discover namespaces, tables, and columns, then catalogs the results in Atlan Cloud.
  • Your REST catalog accepts inbound network connections from Atlan's IP addresses, controlled through your firewall rules or network security groups.
  • In Direct mode, you can browse and select namespaces and tables from the catalog during workflow configuration.

Connect via self-deployed runtime​

A runtime service deployed within your network acts as a secure bridge between Atlan Cloud and your Iceberg REST catalog. This approach works when your catalog must remain fully isolated behind your firewall.

  • A runtime service sits within your network perimeter, deployed on Docker Compose or a Kubernetes cluster.
  • The runtime maintains an outbound HTTPS connection to Atlan Cloud and a local network connection to your Iceberg REST catalog.
  • When you create a crawler workflow, Atlan Cloud sends metadata extraction requests to the runtime through the outbound connection.
  • The runtime translates requests into Iceberg REST API calls, executes them on your catalog, and returns the results to Atlan Cloud.
  • Your REST catalog never exposes ports to the internet—all connections are initiated from within your network.
  • In Agent mode, you must configure catalog details through your secret store since the catalog isn't directly reachable from Atlan Cloud.

How it protects your data​

Iceberg REST catalogs contain metadata about your data lakehouse tables and their locations. Atlan's connection architecture protects your environment through multiple security layers.

  • Metadata extraction, not data replication: Atlan extracts only catalog metadata—namespaces, table schemas, column definitions, and partition information. The actual data files in your object storage (S3, GCS, ADLS) remain in those systems. For example, if your Iceberg catalog has a sales table with 1 billion rows, Atlan discovers the table schema and metadata, but never accesses or stores the actual sales data.

  • Read-only operations: All Iceberg REST API calls are read-only. The connector can't create or drop tables, modify schemas, or change any catalog configuration. The authentication credentials you provide control exactly what the connector can access.

  • Credential encryption: REST catalog authentication tokens are encrypted at rest and in transit. In Direct connectivity, Atlan encrypts credentials before storage. In Self-deployed runtime, credentials never leave your network perimeter—the runtime retrieves them from your enterprise-managed secret vaults (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault) only when needed, and Atlan Cloud never receives or stores them.

  • Network isolation with Self-deployed runtime: Your Iceberg REST catalog gains complete network isolation from the internet. The catalog only accepts connections from the runtime within your local network. The runtime itself only makes outbound HTTPS connections to Atlan Cloud, which your network team can control through firewall rules.

See also​