## SDR Orchestrator FAQ

URL: https://docs.atlan.com/product/connections/self-deployed-runtime/faq/orchestrator-faq

> Frequently asked questions about troubleshooting and monitoring the SDR Orchestrator on Docker, Podman, and Kubernetes

# SDR Orchestrator FAQ

Find answers to common questions about troubleshooting and monitoring the SDR Orchestrator.

:::info **Did you know?**
Some capabilities shown here may require additional enablement or licensing. Contact your Atlan representative for details.
:::

## General

### Minimum requirements

**Docker / Podman (VM):** The SDR Orchestrator itself needs 0.5 vCPU, 256 MB RAM, and ~500 MB disk. Each connector app adds 0.5–1 vCPU, 512 MB–1 GB RAM, and ~1–2 GB disk on top of that. For example, a VM running the SDR Orchestrator plus 3 apps needs at least 2 vCPUs, 4 GB RAM, and 20 GB disk.

**Kubernetes:** The SDR pod requests 100m–500m CPU and 256Mi–512Mi memory. Each app pod requires approximately 0.5–1 vCPU and 512Mi–1Gi memory. The Kubernetes scheduler distributes pods across available nodes—no single-host sizing constraint.

Size your VM or cluster based on the total number of apps you plan to run. See [System requirements](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-with-orchestrator/llms.txt) for the full breakdown.

### SDR state storage

**Docker / Podman:**

| Path | Contents | Backup needed? |
|---|---|---|
| `/mnt/config/config.yaml` | Your infrastructure config (read-only mount) | Yes—this is your only config file |
| `/mnt/deployments/` | Generated per-app deployment directories (docker-compose, .env, Dapr components) | No—the SDR regenerates these on deploy |

There's no database or persistent state beyond these two paths. If the VM is lost, re-run the SDR with the same `config.yaml` and Atlan re-deploys the apps automatically.

**Kubernetes:** State is stored in Kubernetes resources (ConfigMap and Secret) created by Helm. These are regenerated from `values.yaml` during install. If the namespace is deleted, re-run `helm install` with the same `values.yaml` and Atlan re-deploys the apps automatically.

### Running multiple SDR instances

Each SDR instance manages apps on **one VM** (or one Kubernetes namespace). You can't run two SDR instances with the same `deployment_name`.

For multiple VMs or clusters, use a unique `deployment_name` per environment (for example, `mycompany-ec2-prod`, `mycompany-ec2-staging`). Each appears as a separate SDR in the Atlan UI under **Settings** > **Self-Deployed Runtimes**.

## Troubleshooting

### Permission denied on socket

If you see `permission denied` when the SDR tries to manage containers:

- **Docker:** Make sure you included `--group-add $(getent group docker | cut -d: -f3)` in the `docker run` command. This gives the SDR permission to use the Docker socket.
- **Podman:** Verify the socket exists at `/run/user/$(id -u)/podman/podman.sock` and that you started the socket with `systemctl --user enable --now podman.socket`.
- **Kubernetes:** Check the SDR ServiceAccount has the correct RBAC permissions: `kubectl get rolebinding -n atlan -l app=atlan-sdr`. Verify the Role and RoleBinding exist and reference the correct ServiceAccount.

### SDR shows as Disconnected in Atlan UI

- Check that outbound HTTPS and gRPC (port 443) aren't blocked by your firewall or security group.
- Verify the `TEMPORAL_HOST` environment variable points to your Atlan Temporal endpoint.
- Check SDR logs:
 - **Docker:** `docker logs atlan-sdr-{deployment_name}`
 - **Podman:** `podman logs atlan-sdr-{deployment_name}`
 - **Kubernetes:** `kubectl logs -n atlan -l app=atlan-sdr`

 Look for connection errors or authentication failures.

### Image pull fails

- **Docker / Podman:** Verify you ran `docker login -u atlanhq` (or `podman login`) with the correct PAT before starting the SDR.
- **Kubernetes:** Verify the `atlan-registry-creds` secret exists and is referenced in `imagePullSecrets`:
 ```bash
 kubectl get secret atlan-registry-creds -n atlan
 ```
 If the secret is missing, create it:
 ```bash
 kubectl create secret docker-registry atlan-registry-creds \
 --namespace atlan \
 --docker-server=docker.io \
 --docker-username=YOUR_USERNAME \
 --docker-password=YOUR_PAT_TOKEN
 ```
- If using a private registry, check that `container_registry.base`, `username`, and `password` are correct in `config.yaml` (Docker/Podman), or `sdr.config.containerRegistryBase` is set correctly in `values.yaml` (Kubernetes).

### Connector crash loop on EC2—`AAF-STR-004` error and IMDSv2 timeout

If a connector container enters a crash loop before it begins listening on port 8000, and you see these in the logs:

- `AAF-STR-004: Failed to upload file` in the Atlan workflow logs
- `timeout waiting for http://169.254.169.254/latest/api/token` in the container logs

> `169.254.169.254` is the AWS Instance Metadata Service (IMDS) link-local address — it is the same on every EC2 instance.

The cause is EC2's default IMDSv2 hop limit of 1. Docker networking adds one extra network hop, so IMDSv2 token requests from inside containers are silently dropped. The AWS SDK retries for ~180 seconds before the container crashes. This affects any SDR connector that uses IAM instance roles—not just MSSQL.

**Resolution:** Increase the hop limit to 2. No reboot is required.

**AWS Console:**
1. Open the EC2 console and select your instance.
2. Go to **Actions** > **Instance settings** > **Modify instance metadata options**.
3. Set **Hop limit** to `2` and click **Save**.

**AWS CLI:**
```bash
aws ec2 modify-instance-metadata-options \
 --instance-id \
 --http-put-response-hop-limit 2 \
 --http-endpoint enabled
```

**Verify the fix** by running this from the EC2 host:
```bash
docker run --rm curlimages/curl curl -s -X PUT \
 http://169.254.169.254/latest/api/token \
 -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
```
A short token string in the response confirms the metadata service is reachable from inside containers. An empty response or timeout means the change hasn't propagated yet—wait a few seconds and retry.

## Monitoring

### Monitoring SDR

**Docker / Podman:**

- **Container status:** `docker ps | grep atlan-sdr` or `podman ps | grep atlan-sdr`
- **Resource usage:**

 ```bash
 # Docker:
 docker stats atlan-sdr-{deployment_name}

 # Podman:
 podman stats atlan-sdr-{deployment_name}
 ```

- **Logs:**

 ```bash
 # Docker:
 docker logs --tail=100 -f atlan-sdr-{deployment_name}

 # Podman:
 podman logs --tail=100 -f atlan-sdr-{deployment_name}
 ```

**Kubernetes:**

- **Pod status:** `kubectl get pods -n atlan -l app=atlan-sdr`
- **Resource usage:** `kubectl top pods -n atlan -l app=atlan-sdr`
- **Logs:** `kubectl logs -n atlan -l app=atlan-sdr --tail=100 -f`
- **All app pods:** `kubectl get pods -n atlan`

The SDR doesn't expose a metrics endpoint. Monitor it externally via container/pod health status and log aggregation.

### Log lines to watch for

| Log message | Meaning |
|---|---|
| `Connected to Temporal` | SDR successfully connected to Atlan—healthy |
| `Worker started` / `K8s SDR worker listening on task_queue` | SDR is polling for tasks—healthy |
| `connection refused` | Can't reach Temporal endpoint—check firewall/DNS |
| `permission denied` | Socket access issue (Docker/Podman) or RBAC issue (Kubernetes) |
| `pull access denied` | Registry auth failed—check `docker login` or `imagePullSecrets` |

## Versioning and rollback

### How do I see which version I'm running?

Each connector app and the SDR Orchestrator are published as immutable, tagged images (for example, `atlanhq/atlan-oracle-app:main-ec46120`). Tags may be commit-based (`main-<sha>`) or semantic (`3.1.4`)—apps are gradually moving to semantic versioning, so both formats can appear during the transition. To see the versions your tenant needs to run, poll the tenant-specific latest-releases endpoint with an [Atlan API token](https://docs.atlan.com/llms/governance/stewardship/manage-api-tokens/llms.txt):

```bash
curl -sS \
 -H "Authorization: Bearer $ATLAN_API_TOKEN" \
 "https://<your-tenant>.atlan.com/api/service/marketplace/latest-releases"
```

The response lists the `version` and source `image` (`repo:tag`) for every app your tenant can run, plus the SDR Orchestrator image itself. Compare these against what you're actually running:

- **Docker / Podman:** `docker ps --format '{{.Image}}'` (or `podman ps --format '{{.Image}}'`)
- **Kubernetes:** `kubectl get pods -n atlan -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'`

Version visibility is tenant-specific by design: different tenants can be on different active release cohorts at the same time, so there is no single public version page.

### How do I roll back to previous version?

Rollback depends on your [lifecycle mode](https://docs.atlan.com/llms/platform/self-deployed-runtime/app-lifecycle-modes/llms.txt):

- **Single-app mode (you own the lifecycle):** point your `docker-compose.yaml` (or Helm `values.yaml`) back to the previous known-good image tag and redeploy—the inverse of the [guided update](https://docs.atlan.com/llms/platform/self-deployed-runtime/update-guided/llms.txt). Because you control the image tag, pinning and rollback are fully self-service.
- **Orchestrator mode (Atlan-managed lifecycle):** version is managed centrally by design—there is no rollback to a prior version and no self-service version control. The orchestrator continuously reconciles each app (and itself) to the current release Atlan publishes for your tenant (approximately hourly), so pinning an older tag locally is reverted on the next reconcile. Remediation is *fix-forward*: if a regression is found (manually or through monitoring), Atlan publishes a corrected release to the marketplace and the orchestrator picks it up on the next reconcile cycle. If you need to pin specific versions or roll back yourself, use single-app mode.

:::note
Centralized version management is intentional in orchestrator mode—it keeps every app on the release Atlan has validated for your tenant, which is the point of running the orchestrator. Remediation is fix-forward (Atlan ships a corrected release the orchestrator rolls forward to), not a rollback to an older image. Choose single-app mode if you require version pinning or self-service rollback.
:::

## See also

- [Install SDR Orchestrator on Docker/Podman](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-with-orchestrator/llms.txt): Step-by-step setup guide for Docker/Podman.
- [Install SDR Orchestrator on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-orchestrator-on-kubernetes/llms.txt): Step-by-step Kubernetes setup guide.
- [SDR Orchestrator](https://docs.atlan.com/llms/platform/self-deployed-runtime/orchestrator-architecture/llms.txt): Components, communication model, and container management.
- [SDR Orchestrator on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/orchestrator-architecture/llms.txt): RBAC, Helm charts, environment variables, and K8s-specific operational details.
- [Deployment and security FAQ](https://docs.atlan.com/llms/platform/self-deployed-runtime/deployment-and-security-faq/llms.txt): General deployment and security questions.

---