Set up AWS Glue

TL;DR

Learn about set up aws glue.

warning

🤓 Who can do this? You will need your AWS Glue Data Catalog administrator to run these commands - you may not have access yourself.

Atlan supports fetching metadata from AWS Glue Data Catalog. If you also want to be able to preview and query the data, you can set up an Amazon Athena connection instead. Prefix all resources you create for Atlan with atlan- and add AWS tags and descriptions to make them easier to identify later.

Create IAM policy

To create an IAM policy with the necessary permissions follow the steps in the AWS Identity and Access Management User Guide.

Create the policy using the following JSON:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "glue:GetTables",
        "glue:GetDatabases",
        "glue:GetTable",
        "glue:GetDatabase",
        "glue:SearchTables",
        "glue:GetTableVersions",
        "glue:GetTableVersion",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetUserDefinedFunctions",
        "glue:GetUserDefinedFunction"
      ],
      "Resource": [
        "arn:aws:glue:<region>:<account_id>:tableVersion/*/*/*",
        "arn:aws:glue:<region>:<account_id>:table/*/*",
        "arn:aws:glue:<region>:<account_id>:catalog",
        "arn:aws:glue:<region>:<account_id>:database/*"
      ]
    }
  ]
}

Replace <region> with the AWS region of your Glue instance.
Replace <account_id> with your account ID.

The policy is region-scoped - and needs all four Resource lines

The Glue Data Catalog is per-region: if the policy names us-east-1 but the Atlan connection's Region field says us-west-2, calls are denied - or silently return nothing. Keep all four Resource lines (catalog, database/*, table/*/*, tableVersion/*/*/*) - dropping the catalog line breaks glue:GetTables even when the rest is correct. Note that missing permissions often produce empty results instead of errors - see Catalog permissions & Lake Formation.

warning

If you're using AWS Lake Formation to manage access to your AWS resources, you need to grant permissions in AWS Lake Formation and to the objects you want to crawl: DESCRIBE and SELECT on each governed database and table, lakeformation:GetDataAccess, and - if column-level security is configured - access to all columns for the Atlan identity.

Missing Lake Formation grants don't raise errors: the APIs silently return tables with empty column lists, and the crawl can then trip Atlan's circuit breaker (Delete percentage is more than 80.0). Verify up front with the 2-minute aws glue get-tables self-check in Catalog permissions & Lake Formation.

info

If you plan to crawl assets from an Amazon S3 Table Bucket federated catalog, no additional IAM permissions are required. The existing glue:GetDatabases and glue:GetTables permissions apply to federated catalogs when a Catalog ID is configured.

Choose authentication mechanism

Using the IAM policy created in Create IAM policy step, configure one of the following options for authentication.

User-based authentication

To configure user-based authentication:

Create an AWS IAM user by following the steps in the AWS Identity and Access Management User Guide.
On the Set permissions page, attach the policy created in the previous step to this user.
Once the user is created, view or download the user's access key ID and secret access key.

warning
This will be your only opportunity to view or download the access keys. You will not have access to them again after leaving the user creation screen.

Role delegation-based authentication

For the step-by-step role, trust policy, and Atlan allowlist setup, see Connect with a cross-account IAM role. It covers the full five-phase handshake, including the two-ticket flow with Atlan support and the External ID rules.

Set up credentials for SDR

When you use AWS Glue with Self-Deployed Runtime (SDR), the runtime deployed in your environment makes all AWS API calls. Atlan's infrastructure isn't part of the AWS credential chain—your own IAM identities are the only principals that call the Glue and Lake Formation APIs.

The IAM policy from Create IAM policy defines the required Glue permissions. Configure how the runtime authenticates using one of these options:

STS role (recommended for Lake Formation)
Direct IAM user

Use this option when AWS Lake Formation governs access to your Glue databases and tables. The runtime authenticates as an IAM user with minimal permissions, then assumes a separate role that holds the Glue and Lake Formation grants.

Create an IAM user (for example, app-atlan-sdr-crawler). Attach only the permission to call sts:AssumeRole on the target role—don't attach the Glue IAM policy to this user directly.
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<account-id>:role/<role-name>"
    }
  ]
}
```
Replace <account-id> with your AWS account ID and <role-name> with the name of the role you create in the next step.
Create an IAM role (for example, app-atlan-glue-catalog-read) and attach the Glue IAM policy from Create IAM policy to it. If you use Lake Formation, also grant this role the required Lake Formation permissions on the specific databases and tables in scope.

Set the role's trust policy to trust the IAM user's ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<account-id>:user/<iam-user-name>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Replace <account-id> and <iam-user-name> with your AWS account ID and IAM user name. No external ID is required—the trust boundary stays within your AWS account.

View or download the IAM user's access key ID and secret access key.

warning
This is your only opportunity to view or download the access keys. Save them before leaving the user creation screen.
Store the access key and secret key in your secret store. For guidance, see Configure workflow execution.
When configuring the Glue crawler workflow in Atlan, set:
- AWS Access Key and AWS Secret Key to the IAM user credentials from your secret store
- AWS Role ARN to the full role ARN (for example, arn:aws:iam::<account-id>:role/app-atlan-glue-catalog-read)

warning

If you use AWS Lake Formation to govern access, IAM permissions alone aren't sufficient. The calling identity—the assumed role (STS option) or the IAM user (direct option)—also needs Lake Formation grants on the specific databases and tables in scope. Without these grants, IAM validation passes but metadata extraction fails at the Lake Formation layer.

Next steps

Crawl AWS Glue: Configure and run the crawler to extract metadata from AWS Glue

Create IAM policy​

Choose authentication mechanism​

User-based authentication​

Role delegation-based authentication​

Set up credentials for SDR​

Next steps​

Create IAM policy

Choose authentication mechanism

User-based authentication

Role delegation-based authentication

Set up credentials for SDR

Next steps