Skip to main content

Set up AWS PrivateLink to PostgreSQL

TL;DR

Connect Atlan to a private RDS or Aurora PostgreSQL database over AWS PrivateLink, so traffic never crosses the public internet. Five phases, each labeled with who performs it.

Connect

AWS PrivateLink connects Atlan to a private RDS or Aurora PostgreSQL database so traffic never crosses the public internet. Unlike Snowflake-style sources, you build the service side yourself - an NLB and an endpoint service in front of your database. Five phases, each labeled with who performs it.

Who can do this?

You will need your AWS administrator involved - you may not have access to run these tasks yourself.

New to PrivateLink?

Read How database connectivity works first. It explains the plug-and-socket model, the VPCE DNS name, and why pg_hba.conf still applies on the private path.

Prerequisites​

You should already have the following:

  • Your own non-default VPC configured in AWS, with private subnets in the availability zones you'll use.
  • A PostgreSQL RDS instance running in AWS, linked to the non-default VPC.
  • Atlan's AWS account ID - it is not published; request it from support now if you don't have it. Ask in the same ticket you use for the rest of this setup.
  • Someone with AWS console access to create load balancers and endpoint services (cloud/network team), and a Postgres admin for users and grants.

Five phases at glance​

PhaseWhat happensOwnerTypical time
1. PrepareService user + grants; optional (recommended) RDS proxyPostgres admin + cloud team30-60 minutes
2. Build the socketInternal NLB → target group → endpoint service; allow Atlan's principalYour cloud/network team1-2 hours
3. Atlan builds the plugAtlan creates its VPC endpoint targeting your serviceAtlan support1-2 business days
4. Accept and receive DNSYou accept the pending connection; Atlan sends the VPCE DNS nameYour cloud/network team + Atlan support30 minutes
5. Connect and verifyHost = VPCE DNS name; configure auth + SSL; testAtlan User30 minutes

Phase 1 - Prepare the database side​

Owner: Postgres admin + your cloud/network team

Create service user and grants​

Create the service user and grants exactly as in Phase 1 of the internet guide.

Copy network settings​

From within AWS:

  1. Navigate to Services, then Database, then RDS.
  2. On the left, under Amazon RDS, click on Databases.
  3. From the Databases table, click your instance's name under the DB identifier column.
  4. Under the Connectivity & security tab, copy the following values:
    1. Endpoint and Port values
    2. VPC value
    3. Subnet group value
  5. On the left, click Subnet groups.
  6. From the table, click the row whose Name matches the subnet group copied above.
  7. From the Subnets table, copy each value under the CIDR block column for private subnets.

Create inbound rule​

To create an inbound rule allowing your private subnet access to your RDS instance:

  1. On the left, under Amazon RDS, click on Databases.
  2. From the Databases table, click your instance's name under the DB identifier column.
  3. Under the Connectivity & security tab, under the Security column and the VPC security groups heading click the link to your security group.
  4. At the bottom of the screen, change to the Inbound rules tab and click the Edit inbound rules button.
  5. At the bottom of the table, click the Add rule button and create the following rule:
    1. For Type use PostgreSQL if you are using the default port (5432), or use Custom and enter your port under Port range.
    2. For Source use Custom and enter your CIDR range (see Copy network settings).
    3. Repeat these sub-steps for each of your CIDR ranges.
  6. Below the table, click the Save rules button.
Why "optional" understates it

The NLB you build in Phase 2 forwards to an IP address. A bare RDS instance's IP changes on failover and some maintenance - when it does, the NLB silently forwards into the void and crawls start timing out even though "nothing changed." An RDS proxy gives the NLB a stable target. If you skip it, put a process in place to refresh the target group after every failover.

Before you create an RDS proxy, ensure that the user created in the RDS database is enabled with basic authentication. This method uses a username and password to connect to the RDS database.

To create an RDS proxy for your RDS instance:

  1. On the left, under Amazon RDS, click on Proxies.
  2. In the upper right of the Proxies table, click the Create proxy button.
  3. Under Proxy configuration enter the following details:
    1. For Engine family select PostgreSQL.
    2. For Proxy identifier enter a meaningful name for your proxy.
  4. Under Target group configuration for Database choose your RDS instance.
  5. Under Authentication for the Secrets Manager secrets:
    • If you have an existing secret for your RDS instance's database credentials, select it from the drop-down.
    • If not, click the Create a new secret link and enter these details in the new tab:
      1. For Secret type select Credentials for Amazon RDS database.
      2. For Credentials enter the Username and Password of the database user.
      3. Under Database select your RDS instance.
      4. At the bottom of the form, click the Next button.
      5. For Secret name enter a name for the secret.
      6. At the bottom of the form, click the Next button.
      7. Leave the automatic secret rotation off and click the Next button.
      8. Review the secret definition and click the Store button.
      9. Return to the tab where you started creating the RDS proxy.
  6. Under Authentication for IAM authentication:
    • If IAM authentication is set to Required, Atlan will use an IAM role to connect to the RDS proxy.
    • If IAM authentication is set to Not Allowed, basic authentication will be enabled. Atlan will use a username and password to connect to the RDS proxy.
  7. Under Connectivity expand the Additional connectivity configuration:
    1. For VPC security group select Choose existing.
    2. For Existing VPC security groups select the security group you edited with the inbound rules above.
  8. At the bottom right of the form, click the Create proxy button.
  9. From the Proxies table, click the link for the proxy you just created.
  10. Under Proxy endpoints section, copy the hostname in the Endpoint column.

Retrieve IP address to target​

From an EC2 instance in your AWS account, run the following command:

nslookup <endpoint>
  • Replace <endpoint> with the fully-qualified endpoint hostname copied from the RDS proxy created above (or your RDS endpoint if you skipped the proxy).

Copy the IP address that comes back from the command, under Non-authoritative answer and to the right of Address.

Phase 2 - Build endpoint service​

Owner: Your cloud/network team

Start creating NLB​

To create an NLB, from within AWS:

  1. Navigate to Services, then Compute, then EC2.
  2. On the left, under Load Balancing, click on Load Balancers.
  3. At the top of the screen, click the Create Load Balancer button.
  4. Under the Network Load Balancer option, click the Create button.
  5. Enter the following Basic configuration settings for the load balancer:
    1. For Load balancer name enter a unique name.
    2. For Scheme select Internal.
    3. For IP address type select IPv4.
  6. Enter the following Network mapping settings for the load balancer:
    1. For VPC select the VPC where the RDS instance is located (see Copy network settings).
    2. For Mappings select the availability zones with private subnets.
  7. Enter the following Listeners and routing settings for the load balancer:
    1. For Port enter 5432 (or the non-default port value from Copy network settings).
    2. For Default action click the Create target group link. This will open the target group creation in a new browser tab.

Create target group​

To create a target group for the NLB:

  1. Enter the following Basic configuration settings for the target group:
    1. For Choose target type select IP addresses.
    2. For Target group name enter a name.
    3. For Port enter 5432 (or the non-default port value from Copy network settings).
    4. For IP address type select IPv4.
    5. For VPC select the VPC where the RDS instance is located (see Copy network settings).
    6. At the bottom of the form, click the Next button.
  2. Enter the following IP addresses settings for the target group:
    1. For Network select the VPC where the RDS instance is located (see Copy network settings).
    2. For IPv4 address enter the IP address returned by the nslookup command (see Retrieve the IP address to target).
    3. For Ports enter 5432 (or the non-default port value from Copy network settings).
    4. At the bottom of the IP addresses section, click the Include as pending below button.
  3. Confirm the following Review targets settings for the target group:
    1. Confirm IP address matches the IP address returned by the nslookup command.
    2. Confirm Port is 5432 (or the non-default port value used by your RDS instance).
  4. At the bottom of the form, click the Create target group button.

Finish creating NLB​

Return to the browser tab where you started the NLB creation, and continue:

  1. Under Listeners and routing, click the refresh arrow to the far right of the Default action drop-down box.
  2. Select the target group you created above in the Default action drop-down.
  3. At the bottom of the form click the Create load balancer button.
  4. In the resulting screen, click the View load balancer button.

Verify target group is healthy​

To verify the target group is healthy:

  1. From the EC2 menu on the left, under Load Balancing click Target Groups.
  2. From the Target groups table, click the link to the target group you created above.
  3. At the bottom of the screen, under the Details tab, check that there is a 1 under both Total targets and Healthy.

Create endpoint service​

To create an endpoint service, from within AWS:

  1. Navigate to Services, then Networking & Content Delivery, then VPC.
  2. From the menu on the left, under Virtual private cloud click Endpoint services.
  3. At the top of the page, click the Create endpoint service button.
  4. Enter the following Endpoint service settings:
    1. For Name enter a meaningful name.
    2. For Load balancer type choose Network.
  5. For Available load balancers select the load balancer you created above.
  6. Enter the following Additional settings:
    1. For Require acceptance for endpoint enable Acceptance required.
    2. For Supported IP address types enable IPv4.
  7. At the bottom of the form, click the Create button.

Allow Atlan account access​

To allow Atlan's account access to the service, from within the endpoint service screen:

  1. At the bottom of the screen, change to the Allow principals tab.
  2. At the top of the Allow principals table, click the Allow principals button.
  3. Under Principals to add and ARN enter the Atlan account ID.
  4. At the bottom of the form, click the Allow principals button.

Notify Atlan support​

Once all of the above steps are complete, contact Atlan support. You will need to provide Atlan support:

  • The name of the endpoint service created above (looks like com.amazonaws.vpce.<region>.vpce-svc-<hash>). Go to Endpoint Services and copy the "Service Name".
  • The RDS proxy or RDS endpoint DNS - if IAM authentication is enabled on your RDS proxy or RDS database, respectively.
Connecting more than one database?

One NLB can serve several databases using one listener port per database (e.g. 5432, 5433, 5434...). Two hard-won rules from support history: (1) every additional listener port must also be opened on Atlan's side - list every port in your support ticket, including ones you add later; (2) setups where the listener port differs from the database port ("port translation") have triggered connector regressions more than once - keep the mapping simple, prefer 5432→5432 where possible, and test each connection after any Atlan-side upgrade. See Special cases.

Phase 3 - Atlan builds its endpoint​

Owner: Atlan support

Atlan creates a security group and a VPC endpoint in its network targeting your endpoint service. Typically 1-2 business days. Nothing to do but wait - but don't skip Phase 4, because until you accept, the connection carries no traffic.

Phase 4 - Accept connection, receive DNS name​

Owner: Your cloud/network team + Atlan support

The endpoint connection moves through this lifecycle:

  1. Created - Atlan builds its VPC endpoint (Phase 3).
  2. Pending acceptance - waiting for you. Connections time out in this state.
  3. Available - you clicked Accept; traffic can flow. Atlan sends you the VPCE DNS name.
  4. Usable - proceed to Phase 5.

To accept the connection request, from within AWS:

  1. Navigate to Services, then Networking & Content Delivery, then VPC.
  2. From the menu on the left, under Virtual private cloud click Endpoint services.
  3. From the Endpoint services table, select the endpoint service you created in Phase 2.
  4. At the bottom of the screen, change to the Endpoint connections tab.
    1. You should see a row in the Endpoint connections table with a State of Pending acceptance.
    2. Select this row, and click the Actions button and then Accept endpoint connection request.
  5. Wait for this to complete, it could take about 30 seconds.

Then Atlan support replies with the endpoint's DNS name, in this format:

vpce-<hash>-<hash>.vpce-svc-<hash>.<region>.vpce.amazonaws.com

Phase 5 - Connect and verify​

Owner: Atlan User

  1. In the connection form, set Host to the VPCE DNS name from Phase 4 - not your RDS endpoint. Your RDS hostname works in your own tools because they are inside your network; Atlan is not. This mix-up is one of the most repeated causes of "timeout after PrivateLink was accepted" support requests.
  2. Set Port to the NLB listener port for this database.
  3. Choose your auth method. For IAM Role: enter the Role ARN whose trust policy allows Atlan's principal (support confirms the exact principal), the database username, and - important for cross-region setups - the AWS Region field, so the IAM token is signed for the right region.
  4. Set SSL mode require if the database enforces SSL (RDS often does). pg_hba.conf logic applies on the private path exactly as on the public one - see Firewalls, pg_hba.conf, and SSL.
  5. Click Test connection, then run preflight checks and crawl PostgreSQL.
IAM Role authentication - the three AWS-side switches

All three must be true, or you'll see PAM/password errors that look like bad credentials:

  • IAM Database Authentication is enabled on the RDS/Aurora cluster
  • The database user has it: GRANT rds_iam TO atlan_user;
  • The IAM role has rds-db:connect permission for that database user and instance

Success checklist​

  • Endpoint connection shows Available (not Pending) in your AWS console
  • Test connection passes with the VPCE DNS name as Host
  • Preflight checks pass
  • A full crawl completes - and completes again after your next RDS failover test (this is what catches IP-pinned target groups)

Any step failing - see Troubleshooting PostgreSQL connectivity.

Next steps​

Crawl PostgreSQL: Configure and run the crawler to extract metadata from PostgreSQL