Skip to main content

Special cases

TL;DR

Amazon S3 special cases - multi-million-object buckets, multiple AWS accounts, Atlan-hosted bucket flows, External ID limitations, and the region field.

Connect

Multi-million-object buckets, multiple AWS accounts, the flows where traffic reverses onto Atlan-hosted buckets, External ID limitations, and the hidden region field.

Very large buckets - use inventory ingestion​

Direct crawling lists objects live and is recommended for fewer than 1 million objects; above that, use inventory-based ingestion (see Crawl S3). Multi-million-object direct crawls can run for days and die on heartbeat timeouts - inventory mode reads nightly S3-generated reports instead of listing objects one by one.

  • One dedicated destination bucket (optionally with a single shared prefix) collects reports from all source buckets - the crawler supports exactly one (see Inventory report structure).
  • Format CSV or Apache Parquet only; select all metadata fields; current version only; encryption disabled (see Set up inventory reports for S3).
  • Your role additionally needs s3:ListBucket, s3:GetObject, s3:SelectObjectContent on the destination bucket - and the destination bucket's policy must not explicitly deny the role.
  • First report takes up to 48 hours to appear after enabling inventory in AWS - schedule the first crawl after that.

Buckets across multiple AWS accounts​

An IAM role lives in one account, so the clean pattern is one role per bucket-owning account, one Atlan connection per role. For each account: run the full role guide (each role needs its own trust policy and its own Atlan-side allowlist entry - put all of them in one support ticket, clearly labeled per account). Alternative for a few stray buckets: keep one role and grant it cross-account access via each foreign bucket's bucket policy - but remember the KMS key in the foreign account needs a grant too.

Atlan-hosted buckets - when direction reverses​

Some features work the other way around: you read or write a bucket that Atlan hosts. This applies to query mining through S3 (uploading query-history files for warehouse miners), some asset export flows, and downloading lineage outputs.

  • The bucket name and the access grant live on Atlan's side - both come from support. Ask in one ticket: "Please share the mining/export bucket name for my tenant and grant role arn:aws:iam::111122223333:role/example-uploader-role read+write on it."
  • Send a role ARN, not an instance-profile ARN - instance profiles wrap a role, and the wrapper won't work in the bucket policy.
  • If writes fail with PutObject ... Access Denied after the grant, the grant is likely read-only - ask support to confirm s3:PutObject is included.

External ID limitations across workflows​

The S3 crawler form supports an External ID. But if you reuse the same role for package workflows, know these gaps:

  • Asset Import cannot send an External ID today - a trust policy requiring one makes it fail at AssumeRole.
  • Platform migrations have regressed role-ARN/External-ID handling on existing connections before. If a long-stable connection suddenly fails at AssumeRole after a platform update, ask support to verify the connection still carries your role ARN and External ID before touching AWS.
  • Pragmatic pattern: use your External-ID-protected role for the crawler, and a separate role without the condition (scoped to the minimum prefix) for package workflows that need one.

Region field - including where it's invisible​

Every S3 bucket is addressed in its home region; a mismatch fails with PermanentRedirect. Two flavors:

  • S3 crawler: the region dropdown simply must match your buckets' region.
  • Offline/S3 extraction for other connectors (PostgreSQL, Microsoft SQL Server, … reading extract files from your bucket): if the S3-region field is hidden or mislabeled in the UI, it silently defaults to us-east-1 and any bucket elsewhere fails with PermanentRedirect. If you hit this and see no region field to fix, it's not you - raise a ticket describing the missing field.

Other environments​

  • AWS GovCloud / China partitions: ARNs use different prefixes (arn:aws-us-gov:…) and cross-partition role assumption doesn't exist - ask your account team about supportability before planning.
  • S3-compatible storage (MinIO, Ceph, on-prem gateways): not the same connector; check the docs for your product or ask support.