Skip to main content
Agent Evaluation Suites give you a repeatable way to verify that an agent still behaves the way you expect after prompt, model, tool, or skill changes. You define evaluations that score responses, then attach them to scenarios on individual agents so each scenario produces a pass/fail result you can review over time. Use evaluation suites to:
  • Define reusable test suites on agents with conversation scenarios.
  • Assert specific behavior, such as the tools and skills that should load during a conversation.
  • Score agent responses with LLM-as-judge evaluations against a passing threshold.

​
Concepts

  • Evaluation — A reusable LLM-as-judge definition. An evaluation has a name, a prompt that tells the model what to score (for example, whether the agent followed the right instructions, avoided hallucinations, or used the correct tone), a passing threshold, and the model used to score.
  • Scenario — A test case attached to a specific agent. A scenario describes a conversation prompt to run, an assertion (for example, that a particular tool or skill should load), and one or more evaluations that score the response.
  • Run — A single execution of a scenario. Each run produces results for the assertion and every attached evaluation, and is recorded in run history for that scenario.

​
Create an evaluation

App Administrators create evaluations once and reuse them across any number of agents and scenarios.
  1. Open Intelligence → Evaluations in your app.
  2. Click New Evaluation in the top-right corner.
  3. Enter the evaluation details:
    • Name — A descriptive name (for example, Hallucination Judge or Tone Check).
    • Prompt — Instructions that tell the model what to score. Be specific about what counts as passing and failing behavior.
    • Passing Threshold — The score the response must meet or exceed to pass.
  4. Choose a Model from the dropdown to act as the judge.
  5. Click Save.
Reuse the same evaluation across multiple agents and scenarios to standardize scoring — for example, a shared hallucination judge or a tone judge that every customer-facing agent should pass.

​
Attach an evaluation to an agent scenario

Scenarios live on the agent and pair a test prompt with the assertions and evaluations that should run against it.
  1. Open Intelligence → Agents and select the agent you want to evaluate.
  2. Open the Scenarios tab on the agent page and click + Scenario.
  3. Configure the scenario:
    • Name and Description — Identify the scenario and what it covers.
    • Prompt — The user message or conversation that drives the scenario.
    • Timeout — How long the scenario is allowed to run before failing.
    • Model — The model the agent should use for this run.
    • Assertion — A required behavior to verify, such as which tool or skill should load.
    • Evaluations — Select one or more evaluations to score the response.
  4. Click Save.

​
Run a scenario and review results

  1. From the scenario configuration page, click Run Scenario.
  2. Watch the live response in the panel and review the assertion result and each evaluation’s score.
  3. Open the Run history section on the same page to compare the latest run against previous runs over time.
Because run history is preserved per scenario, you can use evaluation suites as a regression check: rerun a scenario after changing a prompt, model, tool, or skill to confirm the agent still meets your expectations before promoting the change.