## Special cases

URL: https://docs.atlan.com/apps/connectors/storage/amazon-s3/how-tos/special-cases

> Amazon S3 special cases - multi-million-object buckets, multiple AWS accounts, Atlan-hosted bucket flows, External ID limitations, and the region field.

Multi-million-object buckets, multiple AWS accounts, the flows where traffic reverses onto Atlan-hosted buckets, External ID limitations, and the hidden region field.

## Very large buckets - use inventory ingestion

Direct crawling lists objects live and is recommended for **fewer than 1 million objects**; above that, use **inventory-based ingestion** (see [Crawl S3](https://docs.atlan.com/llms/connectors/amazon-s3/crawl-s3/llms.txt)). Multi-million-object direct crawls can run for days and die on heartbeat timeouts - inventory mode reads nightly S3-generated reports instead of listing objects one by one.

- One **dedicated destination bucket** (optionally with a single shared prefix) collects reports from all source buckets - the crawler supports exactly one (see [Inventory report structure](https://docs.atlan.com/llms/connectors/amazon-s3/inventory-report-structure/llms.txt)).
- Format **CSV or Apache Parquet** only; select **all metadata fields**; **current version only**; encryption **disabled** (see [Set up inventory reports for S3](https://docs.atlan.com/llms/connectors/amazon-s3/set-up-inventory-reports-for-s3/llms.txt)).
- Your role additionally needs `s3:ListBucket`, `s3:GetObject`, `s3:SelectObjectContent` on the destination bucket - and the destination bucket's policy must not explicitly deny the role.
- First report takes up to 48 hours to appear after enabling inventory in AWS - schedule the first crawl after that.

## Buckets across multiple AWS accounts

An IAM role lives in one account, so the clean pattern is **one role per bucket-owning account, one Atlan connection per role**. For each account: run the full [role guide](https://docs.atlan.com/llms/connectors/amazon-s3/set-up-s3-access-with-a-cross-account-iam-role/llms.txt) (each role needs its own trust policy and its own Atlan-side allowlist entry - put all of them in **one** support ticket, clearly labeled per account). Alternative for a few stray buckets: keep one role and grant it cross-account access via each foreign bucket's [bucket policy](https://docs.atlan.com/llms/connectors/amazon-s3/bucket-policies-kms-keys-and-prefix-scoping/llms.txt) - but remember the KMS key in the foreign account needs a grant too.

## Atlan-hosted buckets - when direction reverses

Some features work the other way around: **you** read or write a bucket that **Atlan hosts**. This applies to query mining through S3 (uploading query-history files for warehouse miners), some asset export flows, and downloading lineage outputs.

- The bucket name and the access grant live on **Atlan's side** - both come from support. Ask in one ticket: *"Please share the mining/export bucket name for my tenant and grant role `arn:aws:iam::111122223333:role/example-uploader-role` read+write on it."*
- Send a **role ARN**, not an instance-profile ARN - instance profiles wrap a role, and the wrapper won't work in the bucket policy.
- If writes fail with `PutObject ... Access Denied` after the grant, the grant is likely read-only - ask support to confirm `s3:PutObject` is included.

## External ID limitations across workflows

The S3 crawler form supports an External ID. But if you reuse the same role for **package workflows**, know these gaps:

- **Asset Import** cannot send an External ID today - a trust policy requiring one makes it fail at AssumeRole.
- Platform migrations have regressed role-ARN/External-ID handling on existing connections before. If a long-stable connection suddenly fails at AssumeRole after a platform update, ask support to verify the connection still carries your role ARN and External ID before touching AWS.
- Pragmatic pattern: use your External-ID-protected role for the crawler, and a separate role without the condition (scoped to the minimum prefix) for package workflows that need one.

## Region field - including where it's invisible

Every S3 bucket is addressed in its home region; a mismatch fails with `PermanentRedirect`. Two flavors:

- **S3 crawler:** the region dropdown simply must match your buckets' region.
- **Offline/S3 extraction for other connectors** (PostgreSQL, Microsoft SQL Server, … reading extract files from your bucket): if the S3-region field is hidden or mislabeled in the UI, it silently defaults to `us-east-1` and any bucket elsewhere fails with `PermanentRedirect`. If you hit this and see no region field to fix, it's not you - [raise a ticket](https://docs.atlan.com/support/submit-request) describing the missing field.

## Other environments

- **AWS GovCloud / China partitions:** ARNs use different prefixes (`arn:aws-us-gov:…`) and cross-partition role assumption doesn't exist - ask your account team about supportability before planning.
- **S3-compatible storage** (MinIO, Ceph, on-prem gateways): not the same connector; check the docs for your product or ask support.

---