πŸ“œ Our Manifesto
🧰 Backup & Disaster Recovery
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ Customer Success & Supporty
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ Community

Air-Gapped Deployment

How to create an Atlan air-gapped deployment on AWS

πŸ“œ Prerequisites

You'll need the following to create your air-gapped deployment:

  • A computer with an internet connection and the following tools installed:

    • Kubectl

    • AWS

  • Create an AWS sub-account (optional but recommended). This will both help monitor costs and be more secure, as it will not interfere with production or other workloads running in the existing AWS accounts.

  • Create a user with the following IAM permissions:

    ​IAM User permissions​

πŸ‘€ Note: You can either create a new IAM user or enable an existing user with the above permissions. After stack creation is complete, you can remove these permissions from the IAM user.

πŸ› οΈ A step-by-step guide for an Atlan air-gapped cloud deployment on AWS

STEP 1: Create a new stack

Go to the CloudFormation AWS console, and select the option to "Create a New Stack". For the template source, select Amazon S3 URL.

Here is the Atlan CloudFormation S3 template URL: https://atlan.s3.ap-south-1.amazonaws.com/deploy/marketplace/cloudformation/templates/main.yaml​

Cloud Formation AWS console

STEP 2: Specify the stack details

Fill in the parameters needed by the CloudFormation template to create the resources.

  • Deployment Method: The deployment method for installing the product. Select "Airgapped".

    • Online: The stack will be launched with Internet Gateway and NAT Gateway. The product will be accessible via the internet without using a VPN.

    • Airgapped: The stack will be launched without any Internet Gateway, NAT Gateway, or public endpoints. The cluster won't have any kind of internet access and will be only accessible via VPN. Some of the features won't be available, such as:

      • Slack notifications

      • Chat feature

      • Email notifications (you'll need to configure AWS SES separately)

  • License URL: You can leave this blank, since it is not required in this case.

Advanced configuration (optional)

  • VPC Configuration: Create network resources like VPC, subnets, route table, and security groups. The default options are already filled in.

    • VPC CIDR: This creates a new VPC CIDR block. Ensure that the CIDR range is different from your existing VPC, which might need to be peered with Atlan's VPC. Also, do not overlap the range with any CIDR block assigned to the IP CIDR to be used by the EKS cluster.

    • Subnet CIDRs:

      • Private Subnets: 250 IP addresses

πŸ‘€ Note:

You can leave the public subnet CIDRs blank, since they won't get created with an airgapped deployment.

  • EKS Configuration: Configuration of the EKS control plane deployed with the stack.

    • Launch EKS in Private Subnet: When this is set to "True", the EKS control plane will deployed in private subnets. You can ignore it as it doesn't apply in this case.

    • EKS Cluster IP CIDR: This is the CIDR block to assign Kubernetes service IP addresses. If you don't specify a block, Kubernetes assigns the addresses 172.20.0.0/16 CIDR. We recommend that you specify a block that does not overlap with resources in other networks peered or connected to your VPC. The block must meet the following requirements:

      • Within one of the following private IP address blocks: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 or 198.19.0.0/16

      • Does not overlap with any CIDR block assigned to the VPC that you selected.

      • Netmask between /24 and /12.

      • 10.0.0.0/8 and 192.168.0.0/12 won’t work. Make sure the netmask and CIDR block is between the mentioned ranges.

      • Recommended CIDR range: /16

    • AWS IAM User/Role ARN to Access the EKS Cluster through Kubectl: Enter the ARN value of the IAM user or role to get access to the EKS cluster via kubectl. The user/role deploying the stack will have access to the EKS cluster. Use this to provide access to any other user. Refer to this page for more information.

  • Nodes Configuration: Configurations for the nodes being launched. The nodes are divided into a 70-30 ratio of spot and on-demand capacity.

    • EC2 Instance Type: The instance type for Atlan nodes. t3a.2xlarge is recommended for normal workloads. For large workloads, you can increase this to m5a.4xlarge or m5.4xlarge.

    • Launch Spark Nodes: This sets whether to launch Spark nodes. Set this to "True" if you have Azure ADLS. This is set to "False" by default.

    • Additional Userscript: You can add a Bash script that needs to be executed on every node whenever a new node is added. For this, put the public link to the script in the "Link to additional script to run on nodes while bootstrapping" field. However, this is completely optional and can be left blank.

πŸ‘€ Note: While passing the script link, make sure it is public and accessible over the internet. You can use a signed URL with timeout. Once the stack is created, the script will be copied to the S3 bucket launched via CloudFormation, and it will only be fetched by nodes from S3 instead of the internet.

  • Transit Gateway Configuration: These are the configurations to set up the Transit Gateway with the Atlan VPC. Read this documentation for detailed steps.

Sample Parameter List #1
Sample Parameter List #1

STEP 3: Configure the stack options

After entering the parameters above, click on "Next". You can define optional tags as per your IT or Security compliance guidelines, then click the "Next" button.

Tags

STEP 4: Verify all the details

Click on the two checkboxes, and then click on "Create stack".

Accept

STEP 5: Wait 35-40 minutes for stack creation

If you face any issues, recheck the parameters. Otherwise, reach out to the Atlan support team with the CloudFormation error logs.

STEP 6: Install and set up KOTS

To set up KOTS, you'll need to download two open-source tools:

  • First, you will install a Kubectl plugin (KOTS) that allows you to dynamically apply Atlan configurations into EKS clusters.

  • Second, you will install Kotsadm, an installable admin console for managing Kubernetes appliances, including Atlan. Later in this document, we will refer to the Kotsadm UI as the "Configuration Console".

To get started, visit the download portal link provided to you by the Atlan team, and enter the password.

Enter password

Once you're logged in, you will see download buttons for various tools and files. Click on the following buttons (shown below):

  • "Download license"

  • "Download bundle" (for the latest Kots Admin Console release)

  • "Get download link" (for the latest release)

Now you'll need to run a series of commands. These must be run on the same machine where you want to run Atlan, which must also have access to the Kubernetes cluster created through the Atlan CloudFormation stack.

First, install the kubectl plugin:

curl https://kots.io/install | bash

Ensure that the versions for the installed kots-cli and the kotsadm.tar.gz are the same. You can get the version of kots-cli by running the following command:

kubectl kots version

Second, install kotsadm into the cluster by running the following:

kubectl kots admin-console push-images PATH_TO_kotsadm.tar.gz ECR_REPOSITORY_URL/ECR_REPOSITORY_NAMESPACE \
--registry-username AWS \
--registry-password ECR_TEMP_PASSWORD
kubectl kots install lite/stable-1 --namespace kots --license-file PATH_TO_LICENSE_FILE --airgap-bundle PATH_TO_AIRGAP_BUNDLE --kotsadm-registry ECR_REPOSITORY_URL --kotsadm-namespace ECR_REPOSITORY_NAMESPACE --wait-duration=20m --port-forward=false --registry-username AWS_IAM_USER_ACCESS_KEY --registry-password AWS_IAM_USER_SECRET_KEY atlan

πŸ‘€ Note:

  • ECR_TEMP_PASSWORD: This can be generated with the below command:

    aws ecr get-login-password
  • ECR_REPOSITORY_URL: Use the output value EcrRepositoryUrl from the CloudFormation stack output.

  • ECR_REPOSITORY_NAMESPACE: Use the output value EcrRepositoryNameSpace from the CloudFormation stack output.

  • PATH_TO_LICENSE_FILE: This is the path to the downloaded license on the user system.

  • PATH_TO_AIRGAP_BUNDLE: This is the path to the downloaded Airgap bundle.

  • AWS_IAM_USER_ACCESS_KEY, AWS_IAM_USER_SECRET_KEY: The IAM user should have read and write access to the ECR repository service.

You will be prompted for a password to use to secure the admin console. Record and store the generated password somewhere safe, as you will need it to manage the Atlan application. If you lose this password, you will lose access to the admin console!

After installation is complete, you can serve the admin console on a local machine by running:

kubectl kots admin-console -n kots

This will serve up the admin console at localhost:8800 on the machine running the command.

If you see errors after installation like the ones below, you can ignore them for now.

Lastly, execute the below command in order to get access to the admin console and keep the command running:

kubectl port-forward svc/kotsadm -n kots 8800:3000

STEP 7: Configure the product

After deployment, you will need to enter the product configuration and set up the organization.

First, access http://localhost:8800 in your browser, and enter the admin console password that you created at the time of deployment.

Login

Next, configure the product. Here is the list of parameters and their values:

Environment and domain configuration
  • Deployment Type: Select Production.

  • Deployment Strategy: Select Airgapped.

  • URL of Private Docker Registry: This is the URL of the private Docker registry. You can get this value from the outputs of CloudFormation.

  • Domain Name: This will be the domain where you will access the product.

  • Master password for services: Enter any password β€” minimum of 10 characters with letters and numbers.

  • Release Portal Domain: This is the domain to access the admin console or release portal. You can map it to your own domain or leave it blank.

  • Restrict Product to Certain IP ranges: Enable this to make the product accessible only through the specified IP ranges. You can leave this as the product is already air-gapped.

    ​TLS configuration​

  • Enable TLS: If you want to upload your SSL certificate and key, then enable this. Otherwise, keep it disabled.

  • AWS Bucket Name: This is the bucket that will be used to store the files used internally by the services and the logs of various jobs. You can get its value from the CloudFormation outputs.

  • Bucket Name for backup: This bucket will be used to store all the backups of the cluster. You can get its value from the CloudFormation outputs.

  • Region of backup bucket: This is the region where the stack is deployed. You can get its value from the CloudFormation outputs.

  • IAM role to use to push backups to the bucket: This is the ARN of the IAM role that will be used by the services to access the backup bucket. You can get its value from the CloudFormation outputs.

    ​Cloud configuration​

  • EKS Cluster Name: This is the name of the EKS cluster which was deployed through CloudFormation. You can get its value from the CloudFormation outputs.

  • AWS Region of Deployment: This is the region where the stack is deployed. You can get its value from the CloudFormation outputs.

  • Keycloak Client Secret: This is the client secret that will be used by Keycloak. You can get this value from the CloudFormation outputs.

    ​Helper Jobs​

  • Activate Auto Release: This will create a job that will automatically install the latest updates to the product. As this is an air-gapped installation, keep it disabled.

Once you've completed these steps, scroll down to the bottom of the page and save the configuration.

A new product version has been created. To implement the changes, click on "Go to the new version".

Now the system will perform the preflight check. Once it is successful without any errors, click on "Deploy".

When the deployment of new changes is complete, wait for all K8s pods to be in the running state. You can check the status of running pods on the K8s cluster by using the below command:

kubectl get pods -A

Access the Atlan UI

To access the Atlan UI, users need to create a DNS entry on their DNS server.

First, get the Nginx LoadBalancer URL:

kubectl get svc nginx-ingress-controller -n nginx-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Say that your LoadBalancer URL is a0376b0f5f0f74fcd9ef6aea05cb7e0c-6d16c0434f421af3.elb.ap-south-1.amazonaws.com, and you want to access the product on https://xyz.company.com.

Then create the following CNAME record on your Domain Name Server.

Once the DNS is configured, you can access the product on https://xyz.company.com.

Next, you'll see a form. Fill this form to set up your account, and click "Setup Account".

Setup

Next, log in with the email and password that you used on the setup page.

Login

Congrats! You've successfully created an Atlan air-gapped deployment on AWS πŸŽ‰

If you have any questions or issues, you can always reach out to us at [email protected].