Create your question set Private Preview

Once you've built and refined your context repository, you need a question set before you can run simulations. A question set is a list of business questions, each paired with a verified SQL query that produces the known-correct answer. This is how CES measures whether the model answers correctly.

Prerequisites

Before you begin, make sure:

• You've built and refined your context repository. See Build your context repository and Refine with Chat & build.
• If you're using Databricks, your repository is deployed. See Deploy to Databricks.

Add questions

CES gives you three ways to add questions. Use them in combination: start by generating questions for coverage, then layer in real questions from Insights or your domain expert.

What makes a good question set

Use questions the business actually asks, phrased the way they phrase them, covering simple lookups, aggregations, multi-table joins, and time-window questions. Start with 10 to 20 entries and make sure every question has a verified SQL query a domain expert has confirmed is correct.

Auto-generate

CES drafts an initial question set by reading your semantic model. This is the fastest way to get started. Use it to confirm coverage across the metrics and dimensions your model exposes.

1. In your context repository, click the Simulate tab.
2. Click Auto-generate.

3. Review each suggestion. Accept the ones that look representative, edit the SQL if the logic is off, and discard anything unrealistic.

Generating questions automatically provides broad coverage but doesn't capture business-specific nuance. For the most important queries, layer in questions from Insights or add them manually.

From Insights

Insights surfaces real questions users have asked in natural-language interfaces across your organization, phrased the way they actually phrase them, including abbreviations and domain-specific shorthand.

note

Insights must be enabled on your tenant for this option to appear. See Enable Insights.

1. In your context repository, click the Simulate tab.
2. Click Add from Insights.
3. Select an Insights collection that matches your domain. For example, a "Sales pipeline" collection aggregating questions from BI tools, search, and chat interfaces.
4. CES pairs each question with a proposed verified SQL query based on your semantic model. Review each entry:
   • Accept entries where the proposed SQL is correct.
   • Edit entries where the question is good but the SQL needs adjustment.
   • Discard entries that don't reflect realistic usage.
5. Click Import selected.

Manual

Some questions only your domain expert can write: multi-step calculations, edge-case filters, and business-critical metrics where a silent wrong answer is expensive.

1. In your context repository, click the Simulate tab.

2. Click Add example.

3. Enter the natural-language question:

   What is the month-over-month NRR trend for customers acquired in 2024?

4. Enter the verified SQL query your domain expert confirms produces the correct answer:

   WITH cohort AS (
     SELECT account_id
     FROM dim_accounts
     WHERE DATE_TRUNC('year', first_active_date) = '2024-01-01'
   ),
   monthly_arr AS (
     SELECT
       account_id,
       DATE_TRUNC('month', as_of_date) AS month,
       SUM(arr) AS arr
     FROM fct_arr_monthly
     WHERE account_id IN (SELECT account_id FROM cohort)
     GROUP BY 1, 2
   )
   SELECT month, SUM(arr) / LAG(SUM(arr)) OVER (ORDER BY month) AS nrr
   FROM monthly_arr
   GROUP BY month
   ORDER BY month;

5. Optionally add tags and notes for context, then click Save.

Next steps

• Simulate: run your question set and act on the diagnostics.