Skip to main content

Amazon Redshift runtime on Docker

Install Amazon Redshift Self-Deployed Runtime using Docker.

Prerequisites​

Before you begin, make sure you have:

  • Docker ecosystem (Docker Engine, Docker CLI, Docker Compose).
  • A Docker Hub Personal Access Token (PAT) from Atlan
  • Object storage: AWS S3, Google Cloud Storage, or Azure Blob Storage with read/write permissions
  • Secret store access: AWS Secret Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault with read permissions

Generate client credentials​

OAuth client credentials are required for the Self-Deployed Runtime app deployment to authenticate successfully to the Atlan tenant. Follow these steps to generate client credentials:

  1. Generate the API token by following the steps in API access documentation.

  2. Create client credentials for App authentication using the Atlan API. Replace {{tenant}} with your Atlan tenant name and {{App Name}} with your application identifier:

    curl --location 'https://{{tenant}}.atlan.com/api/service/oauth-clients' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: Bearer <API token>' \
    --data '{
      "displayName": "{{App Name}}-agent-client",
      "description": "Client for agent oauth for {{App Name}}",
      "scopes": ["events-app-permission-scope","temporal-app-permissions-scope"]
    }'

    • Replace <API token> with the token you generated in step 1.

      Example API response:

      {
        "clientId": "oauth-client-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "clientSecret": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "createdAt": "1756112939595",
        "createdBy": "john.doe",
        "description": "Client for agent oauth for {{App Name}}",
        "displayName": "{{App Name}}-agent-client",
        "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "tokenExpirySeconds": 600
      }
  • Save the clientId and clientSecret values securely. You need these for the deployment configuration.

Prepare deployment environment​

Download the Amazon Redshift app deployment package from Atlan's Docker registry. The package contains the container images and configuration templates required to run the app in your environment.

  1. Use the Personal Access Token (PAT) provided by Atlan to authenticate with Docker Hub:

    docker login -u atlanhq
    # When prompted for password, enter the PAT provided by Atlan

    A "Login Succeeded" message confirms successful authentication.

  2. Download the runtime zip:

    curl -O https://atlan-public.s3.amazonaws.com/redshift-app/docker-compose-with-binary.tar.gz
    tar -xvf docker-compose-with-binary.zip

    This command downloads and extracts the deployment configuration files.

  3. Optional Depending on organizational requirements, you may need to replicate images from Docker Hub to a private image repository. The specific steps vary by organization, here's one approach:

    • Pull the required connector image via Docker CLI:

      docker pull atlanhq/atlan-redshift-app:main-8f855a2abcd

      The command requires the same Docker Hub PAT from Atlan support that you used in step 1 for authentication

    • Push the image to your enterprise's registry. Note down the repository name and image tag generated.

Configure config.yaml​

Customize the deployment by modifying the config.yaml file in the extracted directory:

Configure general settings​

Follow these steps to edit the config.yaml file and update the details:

  1. Update Atlan tenant URL and app credentials:

    version: "1.0"
    apps:
      name: "redshift"
      domain: "<tenant-name>.atlan.com"
      client_id: "<client-id>"
      client_secret: "<client-secret>"
      port: 8000
    • Replace <tenant-name> with your Atlan tenant name.
    • Replace <client-id> and <client-secret> that you generated in Generate client credentials section.

Configure object storage​

Self-Deployed Runtime needs a store for reading or writing files. Configure the object storage that matches your environment:

Did you know?

Dapr supports additional objects stores which aren't mentioned below. For more information, see Dapr object store documentation for other configurations.

  1. Locate the objectstore attribute in config.yaml.
  2. Add local file path configuration for Docker volume mounts.
objectstore:
  type: "local"
  root_path: "/data/storage"

Note: Mount a Docker volume to the specified path in your docker-compose.yml file. This is the recommended option for Docker and local development environments where files can be stored on the host filesystem. Example Docker Compose volume mount:

volumes:
  - ./data:/data/storage

Configure secret storage​

Self-Deployed Runtime fetches secrets from a secret store to connect to the source systems. The secret store references are used to configure the workflow. Configure the secret store that aligns with your security infrastructure:

Did you know?

Dapr supports additional secret stores which aren't mentioned below. For more information, see Dapr secret store documentation for other configurations.

  1. Locate the secretstore attribute in config.yaml.
  2. Add local file secret store configuration. For more information, see Local file secret store.
secretstore:
  type: "local.file"
  secrets_file: "/path/to/secrets.json"
  nested_separator: ":"

Note: Create a JSON file at the specified path containing your secrets in key-value format. Set appropriate file permissions and secure the file. This is the recommended secret store for Docker and local development environments.

Deploy runtime​

Follow these steps to deploy secure App:

  1. Generate the required setup files:

    ./atlan-configurator -input ../config.yaml

  2. Start the deployment:

    docker compose up -d

The deployment process typically takes a few minutes to complete depending on factors like network connectivity and image download time.

For more information on Docker Compose commands and configuration, see the Docker Compose documentation.

Verify deployment​

Follow these steps to verify the deployment:

Verify container​

  1. Check container status:

    docker compose ps

    The output appears similar to:

    NAME                IMAGE                                    COMMAND                  SERVICE             CREATED             STATUS              PORTS
    redshift-agent      atlanhq/atlan-redshift-app:main-8f855a2abcd   "/app/entrypoint.sh"   redshift-agent      2 minutes ago       Up 2 minutes

    The container status shows Up indicating it's running successfully.

  2. Verify logs for the running container:

    docker compose logs redshift-agent --tail=50 -f

    Look for these key log messages that confirm successful deployment:

    • Uvicorn running on http://0.0.0.0:8000 - Web server started
    • Workflow engine initialized - Workflow processing ready
    • dapr initialized. Status: Running - Dapr runtime active
    • Starting worker with task queue: atlan-redshift-DEPLOYMENT_NAME - Worker process started
    • GET /server/ready 200 - Health check endpoint responding

Verify via Atlan UI​

Verify secure agent registration in Atlan:

  1. Sign in to your Atlan tenant as an administrator (for example, https://tenant-name.atlan.com).
  2. Navigate to Workflows > Agent.
  3. Search for your deployment name.
  4. Confirm the agent status shows as Active.

Agent registration and status takes a couple of minutes to reflect in Atlan UI

Next steps​