Bucket policies, KMS keys, and prefix scoping

✦TL;DR

The Amazon S3 second doors - bucket policies, KMS key policies, and how to scope Atlan's access down to a single prefix.

||

|Connect

Doors 3 and 4: even a perfectly configured role can be blocked by the bucket's own policy or its encryption key - and here's how to scope Atlan's access down to a single prefix if your security team requires it.

Bucket policies - bucket's own rulebook

Most buckets don't need any bucket-policy change for Atlan: if your role's permission policy allows the actions and the bucket policy doesn't forbid them, access works. You only need to act when:

• The bucket policy contains explicit Deny statements (common in regulated environments: "deny everything except principals X, Y"). One Deny overrides all Allows - add Atlan's role to the exception list of every Deny that matches, or crawls fail with explicit deny in a resource-based policy even though your role has full permissions. Don't forget the S3 Inventory destination bucket - its Deny statements count too.
• The bucket lives in a different AWS account than the role. Cross-account object access needs the bucket policy to allow the role, too:

{
  "Sid": "AllowAtlanCrawlerRead",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/AtlanS3CrawlerRole" },
  "Action": ["s3:ListBucket", "s3:GetObject", "s3:GetObjectTagging"],
  "Resource": [
    "arn:aws:s3:::example-data-bucket",
    "arn:aws:s3:::example-data-bucket/*"
  ]
}

KMS-encrypted buckets - Door 4

If a bucket's default encryption uses a customer-managed KMS key (SSE-KMS), reading objects requires permission on the key as well as the bucket. Symptoms of a missing key grant: bucket listing works, Test connection may pass, but object reads fail with 403 AccessDenied.

Have your AWS admin add Atlan's role to the key policy:

{
  "Sid": "AllowAtlanDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/AtlanS3CrawlerRole" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

• Reading (crawling, imports): kms:Decrypt + kms:DescribeKey.
• Writing (exports, query-mining uploads to a KMS bucket): add kms:GenerateDataKey.
• Buckets using SSE-S3 (Amazon-managed keys, the default) need none of this.

Inventory reports must be unencrypted

Separately from your data buckets: for inventory report delivery, leave encryption disabled - Atlan's S3 crawler requires unencrypted inventory reports (see Set up inventory reports for S3). If your org mandates encryption at rest on all buckets, discuss this constraint with your security team before choosing inventory ingestion.

Prefix scoping - limiting Atlan to part of a bucket

Two independent knobs exist; use either or both:

• Scope the crawl (what Atlan looks at): in the crawler form, set Include Prefix, e.g. processed/2024/. Exclude filters override includes if both match (see Crawl S3).
• Scope the permissions (what Atlan can look at) - the security-team-grade control:

{
  "Effect": "Allow",
  "Action": ["s3:ListBucket"],
  "Resource": "arn:aws:s3:::example-data-bucket",
  "Condition": { "StringLike": { "s3:prefix": ["processed/2024/*"] } }
},
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:GetObjectTagging"],
  "Resource": "arn:aws:s3:::example-data-bucket/processed/2024/*"
}

Prefix pitfalls

• s3:ListAllMyBuckets cannot be prefix- or bucket-scoped - it stays on "Resource": "*" no matter how tight everything else is. It only reveals bucket names, not contents.
• In Atlan form fields, enter bare names and paths - no s3:// prefix, no leading slash. my-bucket + processed/2024/, not s3://my-bucket/processed/2024.
• Multiple buckets in one include filter are pipe-separated: bucket-a|bucket-b.
• Avoid spaces in folder names - the offline/query-mining extraction workflow fails on them (see Mine queries through S3).
• If the permission scope and the crawl scope disagree, you get partial results with scattered 403s in the logs - keep them aligned.