Skip to main content

Amazon Redshift runtime on Kubernetes

Install Amazon Redshift Self-Deployed Runtime App on your Kubernetes cluster.

Prerequisites​

Before you begin, make sure you have:

  • kubectl installed and configured with administrative access to your target Kubernetes cluster
  • Helm version 3.0 or later installed on your deployment machine
  • A Docker Hub Personal Access Token (PAT) from Atlan
  • Object storage: AWS S3, Google Cloud Storage, or Azure Blob Storage with read/write permissions
  • Secret store access: AWS Secret Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault, or Kubernetes Secrets with read permissions

Generate client credentials​

OAuth client credentials are required for the Self-Deployed Runtime app deployment to authenticate successfully to the Atlan tenant. Follow these steps to generate client credentials:

  1. Generate the API token by following the steps in API access documentation.

  2. Create client credentials for App authentication using the Atlan API. Replace {{tenant}} with your Atlan tenant name and {{App Name}} with your application identifier:

    curl --location 'https://{{tenant}}.atlan.com/api/service/oauth-clients' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: <API token>' \
    --data '{
      "displayName": "{{App Name}}-agent-client",
      "description": "Client for agent oauth for {{App Name}}",
      "scopes": ["events-app-permission-scope","temporal-app-permissions-scope"]
    }'
    • Replace <API token> with the token you generated in step 1.

    Example API response:

    {
      "clientId": "oauth-client-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "clientSecret": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "createdAt": "1756112939595",
      "createdBy": "john.doe",
      "description": "Client for agent oauth for {{App Name}}",
      "displayName": "{{App Name}}-agent-client",
      "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "tokenExpirySeconds": 600
    }
  • Save the clientId and clientSecret values securely. You need these for the deployment configuration.

Prepare deployment environment​

Start by setting up the Docker environment and downloading the necessary deployment files.

  1. Use the Personal Access Token (PAT) provided by Atlan to authenticate with Docker Hub:

    docker login -u atlanhq
    # When prompted for password, enter the PAT provided by Atlan

    A "Login Succeeded" message confirms successful authentication.

  2. Download the Helm charts:

    helm pull oci://registry-1.docker.io/atlanhq/redshift-app --version 0.1.0 --untar

    This command downloads the following files:

    ./Chart.yaml
    ./templates
    ./templates/servicemonitor.yaml
    ./templates/deployment.yaml
    ./templates/service.yaml
    ./templates/dapr-components-cm.yaml
    ./templates/hpa.yaml
    ./templates/service-account.yaml
    ./templates/extra-manifests.yaml
    ./templates/_helpers.tpl
    ./values.yaml

  3. Optional Depending on organizational requirements, you may need to replicate images from Docker Hub to a private image repository. The specific steps vary by organization, here's one approach:

    • Pull the required connector image via Docker CLI:

      docker pull atlanhq/atlan-redshift-app:main-20b504dabcd

      The command requires the same Docker Hub PAT from Atlan support that you used in step 1 for authentication

    • Push the image it to your enterprise's registry. Note down the repository name and image tag generated.

Configure values.yaml​

Customize the deployment by modifying the values.yaml file in the redshift-app directory:

Configure general settings​

Follow these steps to edit the values.yaml file and update the details:

  1. Update container image settings if the Kubernetes cluster is configured with an Enterprise specific private image repository.:

    image:
      repository: atlanhq/atlan-redshift-app # Update if using private registry
      tag: main-20b504dabcd # Update if using different tag
      pullPolicy: Always

  2. Update Atlan tenant URL and app credentials:

    global:
    # Base URLs
    atlanBaseUrl: "<tenant-name>.atlan.com"

    # Authentication - credentials from Pre-Requisites
    clientId: "<client-id>"
    clientSecret: "<client-secret>"

  3. Update the name to identify the deployment. This name appears in the Atlan UI when configuring workflows and helps identify this specific App deployment.

    env:
      - name: ATLAN_DEPLOYMENT_NAME
        value: "redshift-k8s-prod" # Replace with your preferred deployment name

Configure object storage​

Self-Deployed Runtime needs a store for reading or writing files. Configure the object storage that matches your environment:

Did you know?

Dapr supports additional objects stores which aren't mentioned below. For more information, see Dapr object store documentation for other configurations.

  1. Locate the objectstore attribute in values.yaml.
  2. Add AWS S3 configuration. For more information, see AWS S3 Binding Spec.
objectstore:
  enabled: true
  spec:
    type: bindings.aws.s3
    version: v1
    metadata:
    - name: accessKey #optional, leave this empty for iam authentication
      value: ""
    - name: secretKey #optional, leave this empty for iam authentication
      value: ""
    - name: bucket #required, name of the bucket where application can write
      value: "<bucket-name>"
    - name: region #required, region of the bucket where application can write
      value: "<bucket-region>"
    - name: forcePathStyle
      value: "true"

Configure secret storage​

Self-Deployed Runtime fetches secrets from a secret store to connect to the source systems. The secret store references are used to configure the workflow. Configure the secret store that aligns with your security infrastructure:

Did you know?

Dapr supports additional secret stores which aren't mentioned below. For more information, see Dapr secret store documentation for other configurations.

  1. Locate the secretstore attribute in values.yaml.
  2. Add AWS Secret Manager configuration. For more information, see AWS Secrets Manager
secretstore:
      enabled: true
      spec:
        type: secretstores.aws.secretmanager
        version: v1
        metadata:
        - name: region # required, region in which secret is hosted
          value: <secret-region>
          # Needed if IAM authentication is not used
        - name: accessKey
          value: ""
        - name: secretKey
          value: ""

Deploy app​

Follow these steps to deploy secure App:

  • Install the helm chart to deploy your app:

    helm install redshift-agent redshift-app -f redshift-app/values.yaml -n NAMESPACE
    • Replace NAMESPACE with your target Kubernetes namespace.

The deployment process typically takes a few minutes to complete depending on factors like Kubernetes cluster resource availability, private image repository download time etc.

Verify deployment​

Follow these steps to verify deployment:

Verify cluster​

  1. Check pod status:

    kubectl get pods -n NAMESPACE

    The output appears similar to:

    NAME                                           READY   STATUS    RESTARTS   AGE
    redshift-agent-redshift-app-5dff95cd85-cvk5z   1/1     Running   0          2m

    The pod status shows Running with 1/1 ready containers.

  2. Verify logs for the running pod

    kubectl logs -n NAMESPACE -l app.kubernetes.io/name=redshift-app --tail=50 -f

    Example application logs: Look for these key log messages that confirm successful deployment:

    • Uvicorn running on http://0.0.0.0:8000 - Web server started
    • Workflow engine initialized - Workflow processing ready
    • dapr initialized. Status: Running - Dapr runtime active
    • Starting worker with task queue: atlan-redshift-DEPLOYMENT_NAME - Worker process started
    • GET /server/ready 200 - Health check endpoint responding

Verify via Atlan UI​

Verify runtime registration in Atlan:

  1. Sign in to your Atlan tenant as an administrator (for example, https://tenant-name.atlan.com).
  2. Navigate to Workflows > Agent.
  3. Search for your deployment name.
  4. Confirm the agent status shows as Active.

Agent registration and status takes a couple of minutes to reflect in Atlan UI

Next steps​