Skip to main content
Community Hub
TL;DR

Simulations in Context Engineering Studio run autogenerated question sets on your context repository to surface where it answers well, where it misses, and the specific descriptions, joins, filters, or assets to add next.

Your AI can read this via Docs MCPInstall MCP →Connect

Run simulations

A simulation runs an autogenerated question set on your context repository and reports which questions pass, which fail, and what's missing. Each failure points to a specific fix: a missing description, a misrouted join, a missing filter, an out-of-scope asset, or a missing synonym. You describe the fix in the chat window, CES applies it, and you re-run until results are stable enough to confirm the repository is accurate and ready to deploy. Simulate is the Test step of the CES workflow.

info

Simulate is an early-stage feature and still evolving. The workflow and results may change as it matures.

Prerequisites​

Before you begin, make sure:

Run simulations​

  1. In Context Engineering Studio, open your context repository and click the Simulate tab.

  2. Check the Test source card. CES generates a question set from your semantic model and catalog metadata, and shows how many questions are ready, which engine they target, and when it generated them.

    You have three options:

    • Run simulation to use the questions as generated.
    • Re-generate to build a fresh set, for example after you've changed the model.
    • Pick a collection to use a Data Exploration collection instead of the generated set.

  3. Read the questions under Test questions before you run anything. Expand a question to see what it asks. Until you run the simulation, the card is marked Not run yet.

  4. Click Run simulation. This might take a few minutes. When it finishes, you see:

    • Overall Score: Percentage of questions answered correctly and number passing.
    • Test Results: Individual results for each question - whether it passed, and an explanation of what went wrong when it didn't.
    • View SQL: Generated SQL for each question to see what the model produced.

  5. Read the per-question diagnostics, not just the aggregate score. Each failing question tells you something specific about what the model is missing.

  6. Fix the failing questions. Update the linked assets or custom instructions from the Overview tab and regenerate, or describe the change in the chat window and let CES apply it.

  7. Re-run the simulation and review the results. Check whether the questions you targeted now pass, and whether any previously passing questions regressed. Adjust and re-run until the results are stable.

    Databricks simulations

    On Databricks, simulations run sequentially because of Genie API rate limits (5 queries per minute per workspace by default). Expect longer run times than on Snowflake for the same question set. Databricks can raise this quota on request.

Once simulations consistently surface only out-of-scope questions and your domain expert is satisfied with the results, your context repository is ready to deploy.

How to read failures​

Each failing question maps to a specific kind of model gap. Use the failure shape to know what to fix:

Paraphrases produce different answers A synonym is missing. Add the alternate phrasing to the relevant column or metric.

Agent chose an adjacent table The preferred asset isn't flagged strongly enough. Boost the canonical asset's description, or downrank the alternate.

Same metric, different numbers under different filters Two competing definitions exist in the model. Reconcile to a single definition or scope each filter explicitly.

Question can't be answered at all An asset or relationship is out of scope. Add the asset to the repository, or accept it as out of scope.

Next steps​