## Crawl S3 assets

URL: https://docs.atlan.com/apps/connectors/storage/amazon-s3/how-tos/crawl-s3

> Configure and run the S3 crawler to catalog your Amazon S3 buckets and objects in Atlan.

# Crawl S3 assets

Catalog your Amazon S3 buckets and objects in Atlan using the **S3 Assets** workflow. This guide walks you through setting up authentication and running your first crawl.

## Prerequisites

Before you begin, make sure the you have:

- Completed [S3 setup](https://docs.atlan.com/llms/connectors/amazon-s3/set-up-s3/llms.txt) with IAM credentials.
- Your AWS credentials (Access Key ID and Secret Access Key, or Role ARN) ready.
- Information about S3 buckets and prefixes you want to catalog.
- Verified that the AWS account is allowlisted to assume the role when using IAM role-based authentication
- Set up the [destination bucket structure](https://docs.atlan.com/llms/connectors/amazon-s3/inventory-report-structure/llms.txt), required only if you plan to use **inventory-based ingestion**.

## Set up workflow

Create a new **S3 Assets** workflow:

1. In the top navigation, click **Marketplace**.
2. Search for **S3 Assets** and select it.
3. Click **Install**.
4. Once installation completes, click **Setup Workflow** on the same tile.

If you navigated away before installation completed, go to **New** > **New Workflow** and select **S3 Assets** to proceed.

### Configure extraction method

Choose how to connect to your S3 environment:

### Direct extraction

1. Select **Direct** for the extraction method.
2. Choose your authentication type:
 - **IAM User**: Enter your Access Key ID and Secret Access Key.
 - **IAM Role**: Enter your Role ARN. If you configured an External ID in your trust policy during [S3 setup](https://docs.atlan.com/llms/connectors/amazon-s3/set-up-s3/llms.txt), enter the same value in the **AWS External ID** field.
3. Select the AWS **Region** where your buckets are located.
4. Select **Test Authentication** to verify the connection.
5. Select **Next**.

### Agent extraction

1. Select **Agent** for the extraction method.
2. Add the secret keys for your secret store configuration.
3. Follow the [Secure Agent configuration guide](https://docs.atlan.com/llms/platform/secure-agent/configure-secure-agent-for-workflow-execution/llms.txt).
4. Select **Next**.

### Choose ingestion method

Select your ingestion method:

- **Direct ingestion**: Recommended for fewer than 1 million objects. This method crawls S3 buckets and objects directly.
- **Inventory ingestion**: Recommended for large-scale use (more than 1 million objects). Uses inventory reports for efficiency.

For inventory ingestion, provide:

- **S3 Bucket Name**: Bucket holding the inventory reports (without the `s3://` prefix).
- **S3 Bucket Prefix**: Prefix used in the report configuration. Include a trailing slash (`/`). Leave empty if no prefix was used.

:::note
The region for the inventory report is picked from the credentials used in the [extraction method](#configure-extraction-method).
:::

### Configure connection

1. Enter a **Connection Name** to identify your S3 environment. 
 Examples: `production-s3`, `analytics-lake`, `raw-data-store`
2. Assign **Connection Admins** to manage access. At least one admin is required.
3. Bucket assets are mandatory to establish relationships with other two asset types. Configure the scope of assets to be ingested. You can choose to ingest either all three supported asset types (`buckets`, `folders`, and `objects`) or a subset of them. 

### Configure bucket filters

Choose which buckets and prefixes to include or exclude. Exclude filters override include filters if both match.

**For a single bucket**:

- **Include Bucket**: Exact bucket name (for example, `my-data-bucket`)
- **Include Prefix**: Specific prefix to crawl (for example, `processed/2024/`)
- Leave all other filters empty.

**For multiple buckets**:

- **Include Bucket**: Regex pattern (for example, `prod-.*|analytics-.*`)
- **Exclude Bucket**: Regex pattern (for example, `.*-temp|.*-backup`)
- **Include Prefix**: Prefixes to include (for example, `data/|reports/`)
- **Exclude Prefix**: Prefixes to exclude (for example, `archive/|tmp/`)

### Run crawler

You can now start cataloging your assets:

- **Run now**: Select **Run** to start a one-time crawl.
- **Schedule runs**: Select **Schedule & Run** to automate recurring crawls.

Monitor crawl progress in the activity log. Once complete, your S3 buckets and objects will appear in Atlan.

## Troubleshooting

- **Permissions**: Confirm all required IAM permissions are set. See the [S3 setup guide](https://docs.atlan.com/llms/connectors/amazon-s3/set-up-s3/llms.txt) for details.

## Need help

- [Contact Atlan support](https://docs.atlan.com/support/submit-request) for integration issues or assistance.

## See also

- [What Atlan crawls from S3](https://docs.atlan.com/llms/connectors/amazon-s3/what-does-atlan-crawl-from-s3/llms.txt): Full list of assets and metadata included in the crawl.

---