In this document, we will give insight into all the components that are created by our CloudFormation templates. Here is the architecture diagram:
Here is a list of components being created:
This is for isolating the Atlan resources. Atlan needs a /20 CIDR range
Each subnet should have at least 50 private IPs. This is where the EKS control plane is deployed.
Each subnet should have at least 250 private IPs. All the nodes and services will be deployed in these subnets.
These are for routing the traffic from the NAT Gateway and Internet Gateway to the subnets.
These only allow traffic on particular ports for the EKS Control Plane and nodes. This allows the nodes to communicate with the control plane and other nodes.
This will get attached to the NAT Gateway.
This enables nodes in a private subnet to connect to the internet or other AWS services, but prevents the internet from initiating a connection with those instances.
This provides internet access to the VPC.
This distributes incoming traffic between the services. This is attached to Kubernetes services.
This is to allow read and write access to the launched S3 bucket.
S3 Bucket Name
S3 Data Bucket
This is used to store all the logs, images, and artifacts generated by the product.
S3 Backup Bucket
This is used for storing Velero backups of the cluster.
IAM Roles Name
Node Instance Role
This role gets attached to all the nodes deployed in the EKS cluster.
Control Plane Role
This role is used by the EKS controlplane.
Validation Lambda Role
This role is used by the Validation Lambda function.
Kubernetes EBS Tagging Role
This role is used by the EBS Tagging Lambda function.
Control Plane Provision Role
This role is used by the Lambda function that provisions the EKS Control Plane.
CopyZips Userscript Role
This role is used by the Copy Userscript Lambda function. This is created only if the userscript URL is provided in the CloudFormation parameters.
GetCallerArn Lambda Role
This role is used by the GetCallerArn Lambda function.
Kube Config Upload Role
This role is used by the Kube Config Upload Lambda function.
Cleanup Loadbalancer Lambda Role
This role is used by the Cleanup Load Balancer Lambda function.
Atlan EKS Cluster Execution Role
This role is required by the private extension of CloudFormation, which is used for deploying the EKS cluster.
Atlan EKS Cluster Log Delivery Role
This role is required by the private extension of CloudFormation, which is used for deploying the EKS cluster to deliver logs to CloudWatch.
Lambda Functions Name
URL Validation Function
This function checks whether the license URL provided is valid or not before creating any resources.
Cal MinMax Node Function
This function calculates the number of minimum and maximum nodes to be used for creating spot and on-demand node groups. The ratio is 70% spot and 30% on demand.
S3 Bucket Transform Function
This function transforms the name to be used for creating the S3 bucket to satisfy the conditions on S3 bucket names.
EBS Tagging Function
This function tags all the EBS volumes created by Kubernetes with specified tags. This helps during cost calculations.
This function is used to wait for a specified time during creation.
Tag ELB Function
This function tags all the ELBs created by Kubernetes with specified tags. This helps during cost calculations.
Kube Manifest Function
This function is used to create any Kubernetes resources using specified Kubernetes manifests.
Kube Get Function
This function is used to fetch the details about any Kubernetes resources running inside the EKS cluster.
Kube Config Upload Function
This function is used to upload the kubeconfig to an S3 bucket.
This function is used to get the ARN value of the calling IAM role/user.
This function is used to download and save the userscript S3 bucket. This is only created when the userscript URL is provided in a CloudFormation parameter.
Cleanup Load Balancers Function
This function is used to clean up the load balancers
Check EKS Resource Roles
This function is used to check if the roles for the EKS private resource registration are already present or not.
EKS Resource Create Function
This function is used to activate the private CloudFormation extension, which will be used to deploy and manage the EKS cluster. Read more about the extension here.
EKS Network Config Update
This function is used to update the EKS API access to private only. This is only created when the private EKS is set as "True" in the CloudFormation parameters.
EKS Control Plane
The EKS control plane to manage all the Kubernetes resources.
Atlan Nodegroup OD
The Kubernetes managed node group where the service pods will be running. This node group has on-demand capacity. The instance type defaults to
Atlan Nodegroup Spot
The Kubernetes managed node group where the service pods will be running. This node group has Spot capacity. The instance type defaults to
Spark Node Group
This is the node group where all the data profiling pods using Spark will be deployed. It is only required if you use Hive, Glue, or Azure ADLS. The instance size defaults to
There are two namespaces created:
There are two Kubernetes services created, which are the load balancer type:
These are to access the product and admin console on https.
This is used by the CloudFront distributions, and they specify the configuration for caching.
Origin Request Policy
This is used by the CloudFront distributions.
While deploying Atlan through CloudFormation, the deployment is in multiple Availability Zones (AZ) to maximize uptime and availability. This is also beneficial if any AZ goes down — we will still have nodes running in other AZs for smooth functioning of the product. Along with that, Atlan can be deployed in any region.
All the services listed in the tables above are billable by AWS except for Kubernetes resources, as they are deployed inside Kubernetes. None of the services are optional. All are mandatory for smooth functioning of the product.
While launching Atlan using CloudFormation, you can configure the EC2 instances to be used as nodes for the EKS cluster. Here are the parameters available for configuring:
Instance Type: You can choose from the available options for instance type to be used for the nodes. This defaults to
t3a.2xlarge for Atlan nodes. The following instance types are supported for Atlan nodes:
Minimum Nodes: You can decide the minimum number of nodes to be active in the Kubernetes cluster. The minimum value that can be used is 3. This defaults to 4 for Atlan nodes.
Maximum Nodes: You can decide the maximum number of nodes to be active in the Kubernetes cluster. The minimum value that can be used is 6. This defaults to 10 for Atlan nodes.
Default EBS Size: You can set the size of the default EBS that gets attached to nodes. This is only for the nodes' internal use. All the PVCs that are created dynamically provision the EBS. This defaults to 20 GB. The minimum size for EBS is 20 GB.
Apart from the Elastic Kubernetes Service, no other AWS managed service is created in CloudFormation. All the databases are deployed inside Kubernetes itself as a container. The databases that are deployed inside Atlan are Postgres and Redis. These are used for internal workings only.
Although most of the resources are restricted to private access only, there are 2 resources that are deployed publically. Here is a list of these public resources:
EKS Control Plane: The EKS Control Plane is deployed in public subnets so that it can be reachable through internet, although the nodes are in private subnets.
CloudFront Distributions: The CloudFront distributions are publically accessible so that the product can be accessed through the internet.