Skip to main content

How to monitor your AI agents with Atlan

Connect docs via MCP

This page covers monitoring AI agents you built yourself, such as LangChain, LlamaIndex, OpenAI Assistants, or custom agents, when they use Atlan as a governed metadata and context layer. To learn about Atlan's built-in context agents, see Understand context agents.

Monitoring production AI agents usually requires two complementary layers:

  • An LLM observability layer for prompts, completions, traces, spans, latency, token cost, and evals.
  • A data-side observability and governance layer for lineage, access controls, metadata trust signals, and metadata change history.

Atlan adds the second layer. It helps you understand what governed data context an agent used, whether that agent was allowed to access it through MCP, and, where lineage exists, how a retrieved field traces back upstream.

Atlan doesn't replace your LLM observability tool. Pair it with tools such as Langfuse, LangSmith, Arize Phoenix, or Braintrust.

Prerequisites​

  • An Atlan workspace with at least one connected source.
  • The Atlan MCP server configured and reachable from your agent runtime.
  • An LLM observability tool already capturing your agent's traces.
  • Access to the assets and metadata you want the agent to use.
  • For provenance tracing, a source and lineage setup that supports the assets or columns your agent retrieves.

Pair Atlan with LLM observability tool​

Use both systems together.

  1. Choose an LLM observability tool such as Langfuse, LangSmith, Arize Phoenix, or Braintrust.
  2. Instrument your agent runtime to emit traces and spans.
  3. Configure the Atlan MCP server as the governed metadata and context surface your agent calls.
  4. Log enough context in the LLM observability layer to identify which Atlan assets, fields, or MCP calls were involved in a run.
  5. Use Atlan to review the metadata context, lineage, and trust signals behind the retrieved assets.
tip

Use a stable run ID in your own agent logs and LLM observability tooling. It makes cross-checking traces, MCP activity, and asset-level review much easier.

What metrics to track for AI agents​

Track both LLM-layer and data-layer signals.

LLM layer:

  • Latency per span
  • Tool-call success rate
  • Token cost per run
  • Groundedness or faithfulness score
  • Hallucination rate

Atlan layer:

  • Lineage coverage for the retrieved assets or columns
  • Access-denied MCP calls
  • Trust and quality signals on the retrieved assets
  • Metadata changes on relevant assets during the investigation window

Trace what data AI agent retrieved​

Use your LLM observability tool to identify what the agent retrieved, then use Atlan to inspect the data context behind it.

  1. Identify the assets or columns returned during the retrieval step.
  2. Open the corresponding asset in Atlan.
  3. Switch to the lineage view.
  4. If the field supports it, expand to column-level lineage to trace it upstream.
  5. Review ownership, certification, and quality signals on the relevant assets.
note

Lineage coverage depends on the connected source and the lineage available in Atlan. Use lineage where it exists rather than assuming every retrieved field has complete column-level coverage.

Enforce access policies on AI agents​

Treat the agent as a least-privilege client.

  1. Decide whether the agent authenticates as an end user through OAuth, or through a service account or API key flow.
  2. Scope that identity to the minimum metadata reach the agent needs.
  3. Configure your agent runtime to use that identity on every MCP call.
  4. Verify that attempts outside the allowed scope are denied.

All MCP tool calls respect Atlan's existing access controls. Users only see or update metadata they're already allowed to access.

tip

For service-account or API-key setups, keep the scope narrow and map the identity to a single intended access pattern. For user-authenticated OAuth setups, the user's normal Atlan permissions apply.

Audit what AI agent touched​

Start with the assets involved in the run.

  1. Use your LLM observability tool to identify the assets, fields, and MCP calls used during the agent run.
  2. Open the affected assets in Atlan and review their lineage, trust signals, and activity history.
  3. Check asset activity history for metadata changes made during the investigation window.
  4. Compare timestamps and retrieved assets with your LLM observability traces.

For metadata changes, asset history is the most reliable place to start. If the change was made using Atlan AI, you may also see the Updated using Atlan AI marker on the affected asset. To review changes across the workspace, see View event logs.

See also​