## Crawl AWS Glue

URL: https://docs.atlan.com/apps/connectors/etl-tools/aws-glue/how-tos/crawl-aws-glue

> Crawl metadata from AWS Glue Data Catalog to catalog jobs, workflows, tables, and data transformations. Extract lineage information after configuring AWS access permissions.

Extract metadata from your [AWS Glue Data Catalog](https://docs.atlan.com/llms/connectors/aws-glue/set-up-aws-glue/llms.txt) into Atlan. After configuring the necessary AWS access permissions, you can crawl jobs, workflows, tables, and data transformations to discover, catalog, and govern your data assets. Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting.

## Prerequisites

Before you begin, make sure you have:
* Configured [AWS Glue access permissions](https://docs.atlan.com/llms/connectors/aws-glue/set-up-aws-glue/llms.txt)
* Admin or connection admin privileges in Atlan
* Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for running workflows

## Create crawler workflow

To crawl metadata from AWS Glue:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Glue Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Choose extraction method

Select your extraction method and configure the necessary credentials for AWS Glue access.

### Direct extraction

Direct extraction connects Atlan directly to your AWS Glue service to crawl metadata.

1. Configure authentication based on the method you set up when [configuring AWS Glue access permissions](https://docs.atlan.com/llms/connectors/aws-glue/set-up-aws-glue/llms.txt):

 **For IAM User authentication:**
 * Enter the **AWS Access Key** you configured
 * Enter the **AWS Secret Key** you configured
 * Enter the **Region** of your AWS Glue deployment

 **For IAM Role authentication:**
 * Set the **AWS Role ARN** to the ARN of the [role you created in your AWS account](https://docs.atlan.com/llms/connectors/aws-glue/set-up-aws-glue/llms.txt)
 * Enter the **Region** of your AWS Glue deployment

2. Click **Test Authentication** to confirm connectivity to AWS Glue.

3. Once successful, at the bottom of the screen, click **Next**.

### Agent extraction

Agent extraction uses Atlan's Secure Agent (Self-Deployed Runtime) to execute metadata extraction within your organization's environment. The agent runs in your network and calls the AWS Glue API from your infrastructure. For an overview of this connectivity pattern, see [How Atlan connects to AWS Glue](https://docs.atlan.com/llms/connectors/aws-glue/how-atlan-connects-to-aws-glue/llms.txt) and [SDR connectivity](https://docs.atlan.com/llms/catalog/connector-capabilities/sdr-connectivity/llms.txt).

Before configuring the crawler:

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
2. Confirm the runtime can reach the AWS Glue API from your network and that [network security](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-network-security/llms.txt) is configured.

To configure the crawler:

1. Configure the AWS Glue data source by adding the secret keys for your secret store based on your authentication method:

 **For IAM User authentication:**
 * Add the secret key for **AWS Access Key**
 * Add the secret key for **AWS Secret Key**
 * Add the secret key for **Region**

 **For IAM Role authentication:**
 * Add the secret key for **AWS Role ARN**
 * (Optional) Add the secret key for **External ID**
 * Add the secret key for **Region**

2. Complete the Secure Agent configuration by following the instructions in [Configure Secure Agent for workflow execution](https://docs.atlan.com/llms/platform/secure-agent/configure-secure-agent-for-workflow-execution/llms.txt).

3. Click **Next** after completing the configuration.

### Configure connection

Complete the connection configuration for your AWS Glue environment:

1. Provide a **Connection Name** that represents your source environment. For example, you might want to use values like `production`, `development`, `gold`, or `analytics`.

2. To change the users able to manage this connection, change the users or groups listed under **Connection Admins**. If you don't specify any user or group, nobody can manage the connection—not even admins.

3. At the bottom of the screen, click **Next** to proceed.

### Configure crawler

Configure the AWS Glue crawler settings to control which assets are included in the metadata extraction. If an asset appears in both the include and exclude filters, the exclude filter takes precedence.

* **Include Metadata**: Select assets you want to include in crawling. This defaults to all assets if none are specified.
* **Exclude Metadata**: Select assets you want to exclude from crawling. This defaults to no assets if none are specified.
* **Catalog ID**: Enter the identifier of the federated catalog you want to crawl. For example, `123456789012:s3tablescatalog/my-bucket`. Use this when connecting to assets stored in [Amazon S3 Table Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html), which AWS introduced in December 2024. These automatically create a separate federated catalog outside the default `AwsDataCatalog`. Leave this field blank to crawl the standard `AwsDataCatalog`. Existing connections aren't affected when this field isn't set.
* **Preflight checks**: Click to validate credentials and IAM permissions before running the crawler.

:::info

This is a new feature. If you're not using Amazon S3 Table Buckets, leave the **Catalog ID** field blank. The connector defaults to `AwsDataCatalog`, which is identical behavior to previous versions.

:::

When using the Agent extraction method or passing filters via the workflow API, use the following structure. Filters are keyed by the Glue catalog name; for the default AWS Glue Data Catalog the catalog name is `AwsDataCatalog`. Database names in the value list correspond to Glue databases (shown as schemas in Atlan).

| Filter parameter | Format | Example | Description |
|------------------|--------|---------|-------------|
| **include-filter** | `{"AwsDataCatalog": ["db1", "db2", ...]}` | `{"AwsDataCatalog": ["analytics", "production"]}` | Only crawl these Glue databases. Use `{}` to include all databases. |
| **exclude-filter** | `{"AwsDataCatalog": ["db1", "db2", ...]}` | `{"AwsDataCatalog": ["temp", "staging"]}` | Don't crawl these databases. Use `{}` to exclude none. |
| **exclude-table-regex** | String (regex pattern) | `"^tmp_.*"` | Exclude tables whose names match this regex. Empty string = no table exclusion. |

## Run crawler

After completing the configuration:

* To run the crawler once, immediately, at the bottom of the screen click **Run**.
* To schedule the crawler to run hourly, daily, weekly or monthly, at the bottom of the screen click **Schedule & Run**.

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

## See also

* [How Atlan connects to AWS Glue](https://docs.atlan.com/llms/connectors/aws-glue/how-atlan-connects-to-aws-glue/llms.txt): Direct and Self-deployed runtime connectivity
* [What does Atlan crawl from AWS Glue](https://docs.atlan.com/llms/connectors/aws-glue/what-does-atlan-crawl-from-aws-glue/llms.txt): AWS Glue assets and metadata that Atlan discovers and catalogs

---