Import glossaries, glossary term links, descriptions, metadata forms, and Glue skills from AWS Glue business context into Atlan during an AWS Glue crawl.
Import AWS Glue business context Private Preview
Importing AWS Glue business context is in private preview and is enabled per tenant. The Import business context option stays hidden in the workflow setup until the feature is enabled for your tenant. It works only in the AWS regions us-east-1, us-east-2, us-west-2, and eu-west-1. To request access, contact your Atlan customer team. Import Glue skills has the same private preview access. It appears under Import business context, and only when that option is Yes.
The AWS Glue business context APIs that Atlan reads are also in preview at AWS, so AWS can change them. For details, see the AWS announcement and the AWS Glue business context user guide.
Atlan can import the business context you maintain in AWS Glue, such as glossaries, glossary term links, descriptions, and metadata forms, and add it to the Glue assets in Atlan. The import runs after the technical crawl, in the same workflow run. Atlan only reads from AWS Glue and never writes anything back.
The import is off by default. Once you crawl AWS Glue with Import business context set to Yes, your Glue tables, views, and columns carry the terms, descriptions, and form values set on them in AWS.
With Import business context on, you can also turn on Import Glue skills to import the markdown files that AWS Glue links to your tables as Atlan Skills. For details, see Import Glue skills.
Prerequisites
Before you import business context, make sure that:
- Business context is turned on for your tenant. Contact your Atlan customer team. The Import business context option stays hidden until it is.
- You're an admin user in Atlan.
- Your AWS Glue connection uses one of the AWS regions in the preview:
us-east-1,us-east-2,us-west-2, oreu-west-1. In any other region, Atlan skips business context and the rest of the crawl runs normally. - Your AWS administrator has added the business context read permissions to the IAM policy used by the connection, as described in the next section.
Grant IAM permissions for business context
Add the following read actions to the IAM policy you created for Atlan, either in the existing statement or in a new one:
{
"Effect": "Allow",
"Action": [
"glue:ListGlossaries",
"glue:ListGlossaryTerms",
"glue:GetGlossaryTerm",
"glue:GetAsset",
"glue:BatchGetIterableForms",
"glue:ListFormTypes",
"glue:GetFormType"
],
"Resource": "*"
}
If any of these actions is missing, Atlan skips business context and the rest of the crawl runs normally. Preflight checks list the missing actions.
The Atlan CloudFormation template doesn't include these permissions yet. If you created the IAM policy from the template, add them to the policy by hand.
Import AWS Glue business context to Atlan
To import business context, either create a new AWS Glue workflow or modify the crawler's configuration of an existing one. On the metadata step of the workflow:
- For Import business context, click Yes.
- Under Preflight checks, click Check to confirm that the region and the IAM permissions support the import. For the messages you may see, see Check your setup with preflight checks.
- Run or schedule the crawler.
For an existing workflow, editing the workflow and saving it with Import business context set to Yes is enough. The next run imports business context.
Once the crawler has completed running, the imported glossaries, terms, descriptions, and form values are on your Glue assets in Atlan. Atlan imports business context only for the tables and views that the crawl includes, so your include and exclude filters apply.
What Atlan imports
| In AWS Glue | In Atlan |
|---|---|
| Glossary | A glossary named <Glue glossary name> (<connection name>), for example Sales (production). The connection name keeps it apart from glossaries in Atlan that have the same name. |
| Glossary term | A term in that glossary. The term description is the Glue long description, or the short description when there's no long description. |
| Glossary term linked to a table, view, or column | The term linked to the same asset in Atlan. |
Description set in Glue business context on a table or view, for example with update-asset | The description of that asset. It takes precedence over the Glue table description. Descriptions that users edit in Atlan are kept separately and aren't overwritten. |
| Customer-defined metadata form type | A custom metadata set named AWS Glue: <form type name>, with the Glue logo. The set is locked, so it's read-only in Atlan, and it applies to tables, views, and columns. |
| Form values on an asset | The values of that custom metadata set on the same asset. |
Atlan skips AWS-managed form types.
How form field types map
Each field in a Glue form type becomes a field in the custom metadata set.
| Glue field type | Atlan field type |
|---|---|
| String | Text |
| Integer | Integer |
| Long | Integer (large) |
| Double | Decimal |
| Boolean | Boolean |
| Timestamp | Date |
| Any other type | Text |
Atlan imports up to 25 form types, and up to 250 fields per form type. Form types beyond those limits are skipped, and the run ends as a partial success. Custom metadata sets never delete or retype fields. If the type of a field changes in Glue, Atlan skips that field.
How changes in Glue are reflected in Atlan
Each run reads business context again, so changes in AWS Glue reach Atlan on the next run:
| Change in AWS Glue | Result in Atlan on the next run |
|---|---|
| A form is removed from an asset | The values from that form are cleared on the asset. |
| A term is unlinked from an asset | The term link is removed from the asset. |
| A term is deleted | The term is archived. |
| A glossary is deleted | The glossary and its terms are archived. |
The connector only adds and removes links to terms from its own glossaries, the ones named <Glue glossary name> (<connection name>). Links to any other term, including terms users link to assets directly in Atlan, are never changed.
Turn off business context import
Set Import business context to No and save the workflow. Glossaries, terms, and term links already imported stay in Atlan. Table and view descriptions go back to the Glue table description on the next run.
Limitations
- Region. Business context is available only in
us-east-1,us-east-2,us-west-2, andeu-west-1during the AWS preview. - Scope. Atlan imports business context for the tables and views that the crawl includes. AWS business context doesn't use Lake Formation, so Atlan reads business context only for tables that the crawl itself can see.
- Federated catalogs. Business context isn't imported yet for tables in federated catalogs, such as S3 Tables. Atlan still crawls these tables as usual, and imports business context for tables in the default AWS Glue Data Catalog.
- Catalog-level descriptions. These aren't imported yet.
- Run time. Atlan makes about two AWS calls per table, one at a time. Crawls of large catalogs take longer with the import turned on.
Import Glue skills
Glue skills are markdown files in Amazon S3 that AWS Glue links to tables. When you turn on Import Glue skills, Atlan imports them as Atlan Skills, with their markdown and lineage from the tables they're linked to. The option is off by default.
Glue skills follow the same limits as business context: they work only in the four AWS regions that business context supports, and a link to a table in a federated catalog doesn't count, because business context skips those tables.
Grant IAM permissions for Glue skills
On top of the business context permissions, add the following to the IAM policy you created for Atlan:
glue:SearchAssets, to list skillss3:GetObjecton the bucket and prefix that hold the skill fileskms:Decrypton the key, only when the skill files use a customer-managed KMS key
Add these statements to the policy:
{
"Effect": "Allow",
"Action": "glue:SearchAssets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket-name>/<prefix>/*"
},
{
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
}
Replace <bucket-name> and <prefix> with the bucket and prefix that hold your skill files. Replace <region>, <account-id>, and <key-id> with the region, AWS account ID, and ID of the KMS key. Leave out the last statement when your skill files don't use a customer-managed key.
The Atlan CloudFormation template doesn't include these permissions either. If you created the IAM policy from the template, add them to the policy by hand.
If the skill files are in another AWS account, that bucket's policy must also let the Atlan IAM role read them, and so must the key policy when the files use a customer-managed key.
Without glue:SearchAssets, Atlan skips Glue skills and keeps the skills it already imported. Without s3:GetObject or kms:Decrypt, Atlan still imports the skills with their links but can't read their markdown. The rest of the crawl runs normally in both cases. Preflight checks report both.
Turn on Glue skills import
To import Glue skills, either create a new AWS Glue workflow or modify the crawler's configuration of an existing one. On the metadata step of the workflow:
- For Import business context, click Yes. The Import Glue skills option appears under it.
- For Import Glue skills, click Yes.
- Under Preflight checks, click Check to confirm that the permissions support the import. Check your setup with preflight checks explains the messages.
- Run or schedule the crawler.
For an existing workflow, editing the workflow and saving it with Import Glue skills set to Yes is enough. The next run imports Glue skills.
What Atlan imports for Glue skills
| In AWS Glue | In Atlan |
|---|---|
| Glue skills linked to the crawled tables of a catalog | A context repository named Glue skills (<catalog ID>), one per catalog, for example Glue skills (AwsDataCatalog). |
| Glue skill | A Skill. Its name is the Glue skill name, or the skill identifier when the skill has no name. It carries the description and the S3 URI of the markdown as its source path. |
| Markdown file of a skill | A skill artifact on the Skill, holding the markdown content. |
| Link between a skill and a table or view | A lineage process from the table or view to the Skill, named <skill name> describes <table name>. |
To find a skill, search for its name in Atlan, or open a linked table and go to the Lineage tab.
The identity of a Skill is its Glue skill identifier, so renaming a skill in Glue keeps the same asset in Atlan.
Atlan cuts markdown over 256 KiB at 256 KiB and adds a note saying the markdown was cut. Atlan imports a skill without content when its URI isn't an s3:// URI.
Which skills Atlan imports
Glue lists skills for the whole AWS account and region, across all catalogs. Atlan imports only the skills linked to a table or view that this connection crawls. A link counts whichever side declared it, the skill or the table.
Atlan skips skills linked only to tables outside the crawl, for example tables your filters exclude or that Lake Formation hides. It never reads the markdown of a skipped skill, and the run logs how many skills it skipped.
How Glue skill changes reach Atlan
Each run reads Glue skills again, so changes reach Atlan on the next run:
| Change | Result in Atlan on the next run |
|---|---|
| A skill is renamed in AWS Glue | The Skill stays the same asset. |
| A skill is deleted in AWS Glue | The Skill is archived. |
| You remove a skill's last link to a crawled table | The Skill is archived on the next run. Link it again and it comes back as the same asset. |
| You turn off Import Glue skills or Import business context | All imported Skills are archived on the next run. Turn it back on, with Import Glue skills on, and they return as the same assets. |
When a read fails, Atlan archives nothing and the run ends as a partial success:
| What fails | What stays in Atlan |
|---|---|
| Business context is skipped or fails | Every imported Skill stays as it was. |
glue:SearchAssets is denied | Every imported Skill stays as it was. |
| One skill can't be read | That Skill keeps its imported copy while the other Skills update. |
| A markdown file can't be read from S3 | The Skill updates and keeps its imported markdown. |
| Atlan can't read a table | That table's Skills stay, because their link might be on it. |
Turn off Glue skills import
Set Import Glue skills to No and save the workflow. The next run archives every imported Skill. Set it back to Yes to restore them. Setting Import business context to No also archives every imported Skill on the next run.
Check your setup with preflight checks
With Import business context set to Yes, preflight checks add two checks. Both always pass, so they never block you from running the crawler. The message tells you what Atlan skips.
| Check | Message | What it means |
|---|---|---|
| Business context region | Business context is available in <region>. | The import can run. |
| Business context region | Business context is not available in <region> during the AWS preview. The rest of the crawl runs as normal. | Atlan skips business context. Use a connection in a supported region to import it. |
| Business context permissions | Business context permissions validated. | The IAM principal can call the business context actions. |
| Business context permissions | Business context will be skipped. Missing IAM permissions: <actions>. The rest of the crawl runs as normal. | Add the listed actions to the IAM policy, then run the crawler again. |
If the region isn't supported or permissions are missing, the technical crawl still completes. The workflow run ends as a partial success and shows the reason business context was skipped.
With Import Glue skills also set to Yes, preflight checks add two more checks. They're advisory and never block the crawler.
| Check | Message | What it means |
|---|---|---|
| Glue skills search assets | Glue skills permissions validated. or Glue skills are readable, but none were found. | Atlan can list Glue skills. |
| Glue skills search assets | Glue skills will be skipped. Missing IAM permission: glue:SearchAssets. The rest of the crawl runs as normal. | Add glue:SearchAssets to the IAM policy, then run the crawler again. |
| Glue skills search assets | Could not check glue:SearchAssets. The crawl will report it if skills cannot be read. The rest of the crawl runs as normal. | Atlan couldn't test the permission. The crawl reports it if it can't read skills. |
| Glue skills S3 read | Skill markdown is readable from S3. | Atlan can read the markdown of a skill. |
| Glue skills S3 read | Skill markdown will be skipped: s3:GetObject is denied on the skill files, or kms:Decrypt on their customer-managed key. Skills still import with their links. | Add s3:GetObject on the skill files, and kms:Decrypt on the key when the files use a customer-managed key. Then run the crawler again. |
| Glue skills S3 read | Could not read a skill file from S3. Skills still import with their links. | The read of one skill file failed. Skills still import with their links. |
| Glue skills S3 read | No S3 skill file to check; skill markdown will be read during the crawl where allowed. | Atlan found no S3 skill file to test. The crawl reads the markdown it has permission to read. |
If Atlan can't read skills or their markdown, the technical crawl still completes and the workflow run ends as a partial success. Markdown that isn't in S3 and markdown that gets cut make no difference to the run status.
See also
- Set up AWS Glue: Create the IAM policy and credentials Atlan uses
- Crawl AWS Glue: Create and run the AWS Glue crawler
- AWS Glue business context user guide: AWS documentation for the feature Atlan reads
- AWS announcement: AWS What's New post for Glue Data Catalog business context
- Preflight checks for AWS Glue: Checks that validate credentials and permissions before a crawl
- What does Atlan crawl from AWS Glue?: AWS Glue properties and how they map to Atlan assets