Enrich metadata at scale and feed context to AI

Connect docs via MCP

Enrich what matters, not everything; pick the mechanism to the metadata type; and sequence enrichment so it actually grounds your AI use cases. This guide helps you scope which assets to enrich, choose between AI agents, playbooks, and the SDK, and answer the recurring overwrite, quality, and security questions. It's for the team standing up metadata enrichment and feeding governed context to AI agents.

Before answering any enrichment question, identify which of the four levels it's really about: most failed answers solve the wrong level:

Methodology: which metadata to capture, in what order (→ Metadata Model Design method below)
Mechanism: how to apply it at scale (→ the mechanism table)
Trust and governance: overwrite rules, accuracy, security (→ the fills-blanks model and badges)
Downstream value: how enriched context reaches users and agents (→ the flywheel and feed-to-AI guidance)

When to use what

Scope: enrich the consumption layer first. Across large catalogs, typically only a small fraction of assets are actively viewed in a given month, and many of those still lack a description. Retrofitting random enrichment across the whole estate costs far more than doing it focused. Prioritize, in order:

The gold / BI-consumption layer—what stakeholders and downstream agents actually consume.
The most-queried SQL assets (last 30 days).
Assets upstream of the most popular BI reports.
Certified assets, quality-connected assets, and assets in a target domain or data product.

Tie every scope decision to an active business need (an agent build, a domain rollout, an onboarding wave), and make the first action concrete—pick one bounded collection and enrich it this session, not "plan a full enrichment program."

When AI or agent consumption is the goal, weight the type of enrichment too: prioritize column-level description coverage on the agent's target assets before investing in longer-form READMEs or lineage-derived context.

Design combinations, not single elements. A description without an owner doesn't answer questions; lineage without quality indicators doesn't build trust. Map each user story to a combination of metadata:

Trust = description + owner + certificate + quality signal
Discovery = description + tags + glossary link + usage
Compliance = tags + access policies + owners + lineage
Understanding = glossary + README + links + examples

Pick the mechanism to the metadata type—never default to one tool:

Metadata to apply	Mechanism	Notes
Free-text to generate (descriptions, READMEs, SQL intelligence)	AI enrichment agents	Grounded in SQL and lineage; scoped by collection
Categorical, rule-based values (owners, tags, certificates, domains, glossary links)	Playbooks	Filter by asset type and qualified name; schedule to apply to net-new assets
Free-text that already exists elsewhere (legacy-catalog descriptions/READMEs)	SDK / programmatic	Migrate existing content; READMEs need a markdown-to-HTML step
A population defined by SQL, exact-match field lists, or cross-source propagation	Automation workflow	Deterministic; proven at very large asset counts

Confirm the current names, availability, and setup requirements of these tools in the Atlan product documentation and maturity reference: several are evolving, and some may require Atlan assistance to enable.

AI fills blanks; it doesn't overwrite human work. AI-generated descriptions are written only where a human-authored description is absent, and human-authored content takes display precedence—so running AI enrichment won't clobber curated work. Every AI-generated description carries a badge so consumers can calibrate trust. Set this expectation up front—before overwrite concerns come up—and confirm the exact description-property model and precedence rules in the product reference rather than relying on a remembered version.

Atlan operates on metadata, not data. It doesn't scan data values, so it can't identify PII by content—only by column-name patterns or an external classifier. AI enrichment reads metadata, lineage, and SQL history, not live data records. Confirm the exact data-access model—and whether any sample-value option exists and how it's retained—in the product reference before making a security commitment.

Context flywheel

Each phase is a prerequisite for the next. Skip enrichment and the context repository is hollow, simulations fail, and you'll wrongly conclude "AI agents aren't production-ready."

Catalog + lineage. Connect sources; run crawlers and miners to capture DDL/DML, query history, and lineage. This is the raw material.
Enrich. Run enrichment on a scoped, high-value collection, grounded in the structural signals from step 1—not on asset names.
Context-engineer. For a specific agent use case, build a context repository: the bounded semantic model for that one use case, capturing dimensions, metrics, allowed filters and joins, and glossary links. Simulate against a golden-question set and gate on accuracy before deploying. Pull a domain-literate subject-matter expert into the build, not just engineers.
Observe + improve. Capture production query traces and thumbs up/down, and feed failures back into the metadata or the repository.

Before pointing any context-engineering tool at an asset set: confirm the assets are crawled, confirm enrichment has run on those specific assets, scope the eval set (both the questions and the exact tables they hit), then build the semantic model. Enabling it on un-enriched assets yields output no better than the warehouse's native schema.

Metadata model design method

Roll out through the Metadata Model Design method—the durable, mechanism-independent approach:

Gather inputs: 3–5 prioritized user stories, their teams and domains, the tools in use, and where critical data lives.
Map each story to a metadata combination (see the combinations listed earlier).
Document the model: one row per story: story → source systems → asset types → metadata elements → priority.
Validate completeness: every story addressed, all source systems included, daily-touched assets covered. Red flags: single-element stories, missing source systems, "enrich everything."
Design enrichment to win discoverability: verified certificates and glossary-term links carry the most search weight.
Define capture standards (description format, ownership roles, tag naming, certificate levels) so delegated enrichment stays consistent.

Phase rollout

Phase the rollout: prepare templates, enrichers, and a completeness score in week 1; enrich highest-priority assets in complete workflows (source → transformation → consumption) over the following weeks; validate with the original story contributors; then scale by domain with weekly score monitoring.

Admin-first for any new AI surface. Roll out conversational AI to admins first to build internal confidence and a demo-ready experience, then expand via personas or a phased rollout.
Manage access with governance workflows, not persona sprawl. Start with a single admin persona (all authenticated users see everything), use owner-based governance workflows for edit control, and revisit view-level personas only once adoption makes filtering search results a real usability win. The intended operating model: a governance admin defines collections and kicks off enrichment; data stewards review the AI output and add on top—stewards can route assets into a collection by tagging them or adding a custom-metadata field.
For PII tagging where Atlan can't read data: separate identification from access control. Identify candidates from column-name patterns or a data-platform-native scanner, require human confirmation before any PII designation is committed, then control access via a purpose that bundles sensitivity tags. A robust automated pattern orchestrates a native scanner (flags candidates) with Atlan (supplies ownership and tag-gap context) and routes confirmation to asset owners.
For source tags, run a two-lane strategy. Audit which source tags are operationally coupled (access policies, chargeback, protection) before touching them. Keep classification and protection tags native to the source and read-only in Atlan; manage business and operational enrichment tags in Atlan, reverse-syncing only those that match existing source tag definitions. Reserve a dedicated session for tag strategy; never bulk-assign or reverse-sync tags without auditing operational coupling first.
For "same data, different user groups": use one shared context repository with persona-specific overlays (question patterns, synonyms, and instructions per role): never a separate repository per persona. Align personas to job roles (engineer, analyst, product owner), since discovery use cases span domains.
To feed governed context to AI agents: a "talk to data" application needs to know which tables hold customer data, which are certified for production, which columns are sensitive, and who owns access. Export the enriched metadata (descriptions, tags, ownership, certification) so the agent reasons over governed context rather than referencing deprecated tables or exposing sensitive data. Route the extraction schema and query mechanics to the product reference—don't hand-roll them here.

Think of the pipeline as fuel and engine: enrichment populates the metadata (descriptions, READMEs, SQL intelligence, linked terms—the fuel); context engineering consumes that metadata to generate a bounded context repository for a specific agent (the engine). A richer catalog directly produces a better context repository—if the context build is disappointing, fix enrichment first. Keep agent scopes bounded per use case, and separate global skills (org-wide rules, tone, brand conventions) from bounded skills (per-use-case procedures): a single flat skill library bleeds context across use cases as it grows.

The tooling here is three interchangeable, composable blocks—don't conflate them: the context build/test interface (where you assemble and simulate the context repository—not the end-user surface), the compute/AI engine (for example, the warehouse's assistant), and the consumption interface (chat tool, IDE, desktop agent via MCP). Match the consumption interface to where users already work: engineers living in the warehouse → the warehouse assistant; business users → a chat or desktop agent. Confirm current tool names and availability in the maturity reference.

Two rollout notes that repeatedly matter: people are far better editors than authors: always give reviewers AI drafts to confirm rather than blank fields to fill; and for context that lives in code (transformation repositories, pipeline configs), a pragmatic interim is to summarize it outside the platform and bulk-import the summaries as descriptions, rather than waiting for native ingestion of unstructured sources.

Common pitfalls

Standing up tooling before the problem is defined. Establish the context gap or user story first.
Treating "enrich everything" as the goal: it causes deferral and wastes effort on the long tail.
Asserting "it won't overwrite" without explaining the mechanism: the anxiety persists until reviewers understand why it won't.
Requiring 100% human verification before any rollout: this recreates the exact bottleneck manual enrichment caused. Verification and rollout overlap.
Sequencing "foundations first, AI later": well-scoped AI enrichment builds the foundations; deferring it delays value by quarters.
Improvising security, model-version, or transport answers: commit to written documentation instead.
Per-domain persona proliferation, or one context repo per persona: use governance workflows for edit control and one shared model with persona overlays.
Running metadata propagation and AI-agent descriptions together: they're substitutes, not complements. Propagation copies an identical (often misleading) description down lineage; an agent reads the transformation logic and describes what actually changed. Prefer agent-generated descriptions.
Diagnosing setup, SDK, or import failures from a verbal description: reproduce it live on a shared screen; most resolve in one session.

Troubleshooting

Diagnose the instance before defending the tool—almost every "quality" complaint traces to setup, not the model.

A collection is empty or assets aren't appearing. Membership recalculates on a schedule from the metadata lakehouse. Confirm the lakehouse sync is complete and a recent crawl has run, and confirm you're looking at the right collection type for the asset type (a SQL-based collection won't show BI assets).
The agent runs but generates 0 descriptions. Most likely every asset already has a description—AI fills only blanks and skips described assets. Confirm the collection contains un-enriched assets.
Agents fail silently. Check the enrichment agents' prerequisites in the product documentation—where the metadata lakehouse is a prerequisite, it must be provisioned and its catalog sync complete before agents can run; coverage tracking can show assets even when the lakehouse feed is empty. For large tenants the sync can take an hour or more after provisioning.
Low coverage after a migration. Audit the migration source files before blaming enrichment. A common root cause is a strict scope filter in the legacy export where most rows are null (not "no"), so the filter silently skips them. Surprisingly-low coverage is usually an upstream ingestion-filter problem, not downstream enrichment.
Low-quality, "C-grade" AI descriptions. Check which collection was used, whether custom instructions were added (they shape tone, terminology, and persona), how many assets actually ran, and whether lineage and BI are populated. The usual cause is running a single table with no custom instructions and sparse lineage. Run more assets, add custom instructions for domain and persona context, and make sure at least source-to-BI lineage exists.
SQL-intelligence enrichment returns an authorization error. The connector likely lacks query-history access. Confirm the appropriate role and that query-log access is enabled on the connector.
You need to roll back AI descriptions. Override a single asset by writing a human description to the standard description field. Confirm the current rollback options and any bulk-rollback path in the product reference before promising behavior.
Descriptions need to stay in sync with a second tool. For keeping descriptions aligned between Atlan and another catalog or modeling tool, use the metadata-propagation utility that matches records by primary key (it may need enablement). Rule out sync-to-source first when your source is CI/CD-managed (for example, dbt): promoted descriptions get overwritten by the next deploy, so the source repo must be the write point in that case.

For custom-metadata scope: custom-metadata sets currently apply broadly across all assets of a type, with no built-in per-domain segmentation—so scope your sets deliberately to the asset types that need them, and confirm the current segmentation capabilities in the product reference.

Route feature names, GA/preview status, prerequisites, credit/cost mechanics, and lakehouse query schemas to the Atlan product documentation and the maturity reference—those change with each release and aren't restated here.

When to use what​

Context flywheel​

Metadata model design method​

Phase rollout​

Common pitfalls​

Troubleshooting​

See also​