Skip to main content
Community Hub
TL;DR

Frequently asked questions about troubleshooting and monitoring the SDR Orchestrator on Docker, Podman, and Kubernetes

Your AI can read this via Docs MCPInstall MCP →Connect

SDR Orchestrator FAQ

Find answers to common questions about troubleshooting and monitoring the SDR Orchestrator.

Did you know?

Some capabilities shown here may require additional enablement or licensing. Contact your Atlan representative for details.

General​

Minimum requirements​

Docker / Podman (VM): The SDR Orchestrator itself needs 0.5 vCPU, 256 MB RAM, and ~500 MB disk. Each connector app adds 0.5–1 vCPU, 512 MB–1 GB RAM, and ~1–2 GB disk on top of that. For example, a VM running the SDR Orchestrator plus 3 apps needs at least 2 vCPUs, 4 GB RAM, and 20 GB disk.

Kubernetes: The SDR pod requests 100m–500m CPU and 256Mi–512Mi memory. Each app pod requires approximately 0.5–1 vCPU and 512Mi–1Gi memory. The Kubernetes scheduler distributes pods across available nodes—no single-host sizing constraint.

Size your VM or cluster based on the total number of apps you plan to run. See System requirements for the full breakdown.

SDR state storage​

Docker / Podman:

PathContentsBackup needed?
/mnt/config/config.yamlYour infrastructure config (read-only mount)Yes—this is your only config file
/mnt/deployments/Generated per-app deployment directories (docker-compose, .env, Dapr components)No—the SDR regenerates these on deploy

There's no database or persistent state beyond these two paths. If the VM is lost, re-run the SDR with the same config.yaml and Atlan re-deploys the apps automatically.

Kubernetes: State is stored in Kubernetes resources (ConfigMap and Secret) created by Helm. These are regenerated from values.yaml during install. If the namespace is deleted, re-run helm install with the same values.yaml and Atlan re-deploys the apps automatically.

Running multiple SDR instances​

Each SDR instance manages apps on one VM (or one Kubernetes namespace). You can't run two SDR instances with the same deployment_name.

For multiple VMs or clusters, use a unique deployment_name per environment (for example, mycompany-ec2-prod, mycompany-ec2-staging). Each appears as a separate SDR in the Atlan UI under Settings > Self-Deployed Runtimes.

Troubleshooting​

Permission denied on socket​

If you see permission denied when the SDR tries to manage containers:

  • Docker: Make sure you included --group-add $(getent group docker | cut -d: -f3) in the docker run command. This gives the SDR permission to use the Docker socket.
  • Podman: Verify the socket exists at /run/user/$(id -u)/podman/podman.sock and that you started the socket with systemctl --user enable --now podman.socket.
  • Kubernetes: Check the SDR ServiceAccount has the correct RBAC permissions: kubectl get rolebinding -n atlan -l app=atlan-sdr. Verify the Role and RoleBinding exist and reference the correct ServiceAccount.

SDR shows as Disconnected in Atlan UI​

  • Check that outbound HTTPS and gRPC (port 443) aren't blocked by your firewall or security group.

  • Verify the TEMPORAL_HOST environment variable points to your Atlan Temporal endpoint.

  • Check SDR logs:

    • Docker: docker logs atlan-sdr-{deployment_name}
    • Podman: podman logs atlan-sdr-{deployment_name}
    • Kubernetes: kubectl logs -n atlan -l app=atlan-sdr

    Look for connection errors or authentication failures.

Image pull fails​

  • Docker / Podman: Verify you ran docker login -u atlanhq (or podman login) with the correct PAT before starting the SDR.
  • Kubernetes: Verify the atlan-registry-creds secret exists and is referenced in imagePullSecrets: 
    kubectl get secret atlan-registry-creds -n atlan
    If the secret is missing, create it: 
    kubectl create secret docker-registry atlan-registry-creds \
      --namespace atlan \
      --docker-server=docker.io \
      --docker-username=YOUR_USERNAME \
      --docker-password=YOUR_PAT_TOKEN
  • If using a private registry, check that container_registry.base, username, and password are correct in config.yaml (Docker/Podman), or sdr.config.containerRegistryBase is set correctly in values.yaml (Kubernetes).

Connector crash loop on EC2—AAF-STR-004 error and IMDSv2 timeout​

If a connector container enters a crash loop before it begins listening on port 8000, and you see these in the logs:

  • AAF-STR-004: Failed to upload file in the Atlan workflow logs
  • timeout waiting for http://169.254.169.254/latest/api/token in the container logs

169.254.169.254 is the AWS Instance Metadata Service (IMDS) link-local address — it is the same on every EC2 instance.

The cause is EC2's default IMDSv2 hop limit of 1. Docker networking adds one extra network hop, so IMDSv2 token requests from inside containers are silently dropped. The AWS SDK retries for ~180 seconds before the container crashes. This affects any SDR connector that uses IAM instance roles—not just MSSQL.

Resolution: Increase the hop limit to 2. No reboot is required.

AWS Console:

  1. Open the EC2 console and select your instance.
  2. Go to Actions > Instance settings > Modify instance metadata options.
  3. Set Hop limit to 2 and click Save.

AWS CLI:

aws ec2 modify-instance-metadata-options \
  --instance-id <YOUR_INSTANCE_ID> \
  --http-put-response-hop-limit 2 \
  --http-endpoint enabled

Verify the fix by running this from the EC2 host:

docker run --rm curlimages/curl curl -s -X PUT \
  http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"

A short token string in the response confirms the metadata service is reachable from inside containers. An empty response or timeout means the change hasn't propagated yet—wait a few seconds and retry.

Connector containers can't reach Atlan when bridge NAT is restricted​

Each connector container gets its own network namespace, so it doesn't inherit the Orchestrator's network path. On hosts where container bridge networking has no working NAT or masquerade rule, the Orchestrator stays healthy and connected while every connector it starts fails to reach your tenant.

Symptoms:

  • The Orchestrator shows as Connected in the Atlan UI, and its logs show Connected to Temporal.

  • Connector containers fail their uploads after roughly three minutes:

    OSError: Generic S3 error: Error performing PUT
    https://<your-tenant>.atlan.com/api/blobstorage/... after 5 retries, 180s
    - HTTP error: error sending request

  • curl to your tenant succeeds from the host but times out from inside a container.

warning

Connectors reach Temporal, authentication, and blobstorage over the same port 443 path. When that path is broken, metadata extraction is blocked too, not only observability uploads. Check whether the connector's workflows run at all before treating this as an upload-only problem.

Confirm the diagnosis by testing egress from inside a container on the same network the connectors use:

# Podman:
podman run --rm --network podman curlimages/curl:8.10.1 \
  -sv --max-time 15 https://<your-tenant>.atlan.com/api/blobstorage -o /dev/null

# Docker:
docker run --rm --network bridge curlimages/curl:8.10.1 \
  -sv --max-time 15 https://<your-tenant>.atlan.com/api/blobstorage -o /dev/null

When the host reaches your tenant and this command times out, container egress is the problem.

Recommended approach: restore egress for the container network on the host. Work through these in order, because the first two account for most cases on rootful Podman:

  1. Firewall reload flushed the container NAT rules. Any firewall-cmd --reload, including one triggered by configuration management, drops the rules the container engine installs, and containers lose outbound connectivity until those rules are rewritten. Check for a masquerade rule covering your bridge subnet:

    sudo nft list ruleset | grep -i netavark

    To restore the rules and keep the bridge interface trusted:

    sudo firewall-cmd --permanent --zone=trusted --add-interface=podman0
    sudo firewall-cmd --reload
    sudo systemctl restart podman

    The Orchestrator redeploys the connectors on its next reconcile cycle.

  2. Bridge subnet overlaps an internal route. Compare the container network subnet against the host routing table:

    podman network inspect podman | grep -i subnet
    ip route

    When the default subnet overlaps a route on your network, return traffic goes to the wrong next hop and presents as a timeout. Recreate the container network on a free subnet.

  3. Forwarding is disabled or blocked. sysctl net.ipv4.ip_forward returns 1 on a working host. Also check for a FORWARD chain policy of DROP with no container ACCEPT rules, which is common on hardened Linux builds.

  4. MTU mismatch. On a VM behind a VPN or overlay network with an MTU below 1500, the TCP handshake succeeds and the TLS upload then stalls, which matches the 180-second timeout in the error. Recreate the network with --opt mtu=1400 to test this.

  5. Unusable IPv6 route. Re-run the verification command with --ipv4. When it succeeds only with that flag, the container namespace is selecting an IPv6 route the host can't use.

If you can't change host firewall policy, two alternatives keep the Orchestrator unchanged:

  • Point the default container network at a working one. Podman resolves the bridge network mode to whatever its default network is. Create a network with a working subnet and MTU, then set it as default_network under [network] in /etc/containers/containers.conf. The Orchestrator continues to request bridge and gets your corrected network, with no change to config.yaml. Validate this on one deployment before you apply it broadly.
  • Route egress through a proxy. When your environment reaches Atlan through a forward proxy rather than direct NAT, set the proxy variables on the Orchestrator once—it passes its environment to every connector it starts. See Route egress through a proxy.

Host networking for connector containers isn't supported. network_mode isn't a valid key under container_host_config in config.yaml. The Orchestrator accepts only the keys in this table and discards anything else, logging Ignoring unsupported container_host_config keys:

KeyPurpose
dnsCustom DNS servers for connector containers
dns_searchDNS search domains
extra_hostsStatic hostname-to-IP mappings
shm_sizeShared memory size

Connector containers run on the default bridge network so that each one publishes its application port to its own host port. Under host networking there's no port mapping, and every connector binds the same host port directly, which limits the host to a single running connector. Restore container egress on the host instead.

Monitoring​

Monitoring SDR​

Docker / Podman:

  • Container status: docker ps | grep atlan-sdr or podman ps | grep atlan-sdr

  • Resource usage:

    # Docker:
    docker stats atlan-sdr-{deployment_name}

    # Podman:
    podman stats atlan-sdr-{deployment_name}

  • Logs:

    # Docker:
    docker logs --tail=100 -f atlan-sdr-{deployment_name}

    # Podman:
    podman logs --tail=100 -f atlan-sdr-{deployment_name}

Kubernetes:

  • Pod status: kubectl get pods -n atlan -l app=atlan-sdr
  • Resource usage: kubectl top pods -n atlan -l app=atlan-sdr
  • Logs: kubectl logs -n atlan -l app=atlan-sdr --tail=100 -f
  • All app pods: kubectl get pods -n atlan

The SDR doesn't expose a metrics endpoint. Monitor it externally via container/pod health status and log aggregation.

Log lines to watch for​

Log messageMeaning
Connected to TemporalSDR successfully connected to Atlan—healthy
Worker started / K8s SDR worker listening on task_queueSDR is polling for tasks—healthy
connection refusedCan't reach Temporal endpoint—check firewall/DNS
permission deniedSocket access issue (Docker/Podman) or RBAC issue (Kubernetes)
pull access deniedRegistry auth failed—check docker login or imagePullSecrets

Versioning and rollback​

How do I see which version I'm running?​

Each connector app and the SDR Orchestrator are published as immutable, tagged images (for example, atlanhq/atlan-oracle-app:main-ec46120). Tags may be commit-based (main-<sha>) or semantic (3.1.4)—apps are gradually moving to semantic versioning, so both formats can appear during the transition. For the steps to find the version your tenant is assigned, see Find exact version assigned to your tenant.

Compare that against what you're actually running:

  • Docker / Podman: docker ps --format '{{.Image}}' (or podman ps --format '{{.Image}}')
  • Kubernetes: kubectl get pods -n atlan -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'

Version visibility is tenant-specific by design: different tenants can be on different active release cohorts at the same time, so there is no single public version page.

How do I roll back to previous version?​

Rollback depends on your lifecycle mode:

  • Single-app mode (you own the lifecycle): point your docker-compose.yaml (or Helm values.yaml) back to the previous known-good image tag and redeploy—the inverse of the guided update. Because you control the image tag, pinning and rollback are fully self-service.
  • Orchestrator mode (Atlan-managed lifecycle): version is managed centrally by design—there is no rollback to a prior version and no self-service version control. The orchestrator continuously reconciles each app (and itself) to the current release Atlan publishes for your tenant (approximately hourly), so pinning an older tag locally is reverted on the next reconcile. Remediation is fix-forward: if a regression is found (manually or through monitoring), Atlan publishes a corrected release to the marketplace and the orchestrator picks it up on the next reconcile cycle. If you need to pin specific versions or roll back yourself, use single-app mode.
note

Centralized version management is intentional in orchestrator mode—it keeps every app on the release Atlan has validated for your tenant, which is the point of running the orchestrator. Remediation is fix-forward (Atlan ships a corrected release the orchestrator rolls forward to), not a rollback to an older image. Choose single-app mode if you require version pinning or self-service rollback.

See also​