Skip to main content

Special cases

TL;DR

AWS Glue special cases - multiple AWS accounts, multiple regions, cross-account resource links, S3 Table Bucket federated catalogs, and the circuit breaker.

Connect

Multiple AWS accounts, multiple regions, catalogs shared across accounts, federated S3 Table Bucket catalogs, and how the circuit breaker behaves when the source changes underneath you.

Multiple AWS accounts​

Each Glue Data Catalog lives in one AWS account - so each account needs its own connection in Atlan, its own IAM role (or user), and - for the role method - its own allowlist entry on Atlan's side. This is a very common enterprise pattern (dev/UAT/prod accounts, or one account per data domain). To set up several at once:

  • Raise one support ticket listing every environment: tenant URL, role ARN, and AWS account ID per row, clearly labeled. Support returns the matching node instance role ARN per tenant and allowlists everything in one pass.
  • Double-check the tenant ↔ account mapping. Multi-environment mix-ups (dev tenant ARN in the prod trust policy, or vice versa) cause repeated failed tests.
  • Adding an account later? That's a new allowlist entry - a quick ticket with the new role ARN and account ID.

Multiple regions​

The Glue Data Catalog is region-scoped. One Atlan connection crawls one region - the Region field in the connection decides which catalog Atlan sees.

  • Catalogs in us-east-1 and eu-west-1 of the same account = two connections in Atlan.
  • The IAM policy's Resource ARNs name a region: make sure they cover every region you connect (repeat the resource lines per region, or use one policy per role per region).
  • A region mismatch between the policy, the credentials, and the connection's Region field produces either an authentication error or - worse - an empty catalog and a tripped circuit breaker. When a connection that worked suddenly returns nothing, confirm nobody changed the Region field or moved the catalog.
  • Your Atlan tenant's own region doesn't need to match your catalog's region - cross-region API calls are how the connector normally works.

Glue lets an account "link" databases owned by a different account. The catch: permissions on a resource link don't carry across accounts. The Atlan role in account A can list the linked database, but calling GetTables on it's authorized by account B - the owner.

  • Symptom: most databases crawl fine, but the linked ones show 0 tables, or logs show AccessDeniedException … on resource: arn:aws:glue:<region>:<other-account-id>:catalog naming an account ID that isn't the one you configured.
  • Fix: the owning account must grant glue:GetTables / glue:GetDatabases (and Lake Formation grants, if used there) to the Atlan role's ARN.
  • The partial visibility also means each failed run can trip the circuit breaker if the missing databases were previously crawled successfully.

Amazon S3 table buckets (federated catalogs)​

Tables in Amazon S3 Table Buckets surface through a federated catalog. No extra IAM permissions are required, and these tables appear in Atlan with an Iceberg Table subtype label - that's expected, they behave as standard Glue assets. Enter the federated catalog's identifier in the Catalog ID field of the crawler configuration. See Set up AWS Glue.

Iceberg, Delta, and externally managed schemas​

If your tables' schemas are managed outside the Glue table definition (Iceberg/Delta with schema-on-read, or an external Hive metastore), a Glue ETL job or AWS-side crawler re-run can overwrite table definitions without column info. Atlan then correctly extracts tables with 0 columns - and the circuit breaker stops the publish. If columns vanish, check the AWS-side job history (CloudTrail: UpdateTable / CreateTable) before suspecting the connector.

Circuit breaker, briefly​

Atlan compares each crawl against what it already cataloged. If a crawl deletes more than 80% of existing assets, the workflow stops with Delete percentage is more than 80.0. Exiting. - publishing nothing rather than wrongly archiving your catalog.

  • It's a symptom, not the disease. Something made the source return far less: permissions, Lake Formation, a region change, or a genuine cleanup in Glue.
  • Genuine cleanup? If you really did delete or rename most databases (for example, moving to a new naming convention), the breaker is doing its job; support can help the workflow accept the intentional change.
  • Fix the source-side cause and the next run clears automatically.