Test Case detail
Open a single test to see what it expects, how it has done lately, and what the agent did when the test was captured. This is where you decide whether a test still earns its place in your suite, and fix it when it doesn't. Click any test in the Testing Lab to open it.
At the top you'll see the test's type, its latest result - Passing, Failing, Systematic error or Never run - the agent it targets, how many messages it has, and its pass rate across its runs. Next to them are Run test, Edit and Delete. Below are two tabs, Overview and Conversation.
Overview
| Section | What it's for |
|---|---|
| Expected outcome | What a pass looks like, in plain words. Edit it in place when the requirement changes. |
| Datasets | Which datasets include the test. Add it to another one, or remove it from one. |
| Properties | The Agent the test targets, and its Category. |
| Recent runs | The test's latest results. Click one to open that run's report, or View all to see every run of this test in Run history. |
A pass rate that swings between runs is worth a look before anything else. Either the agent wavers - tighten its instructions - or the test is unclear about what counts as right. See Read a run.
Conversation tab - inline traces
If the test went through Run preview when it was created, each agent reply is followed by Trace events: what the agent did on that turn to produce it.
- Tool call and Result - which tool the agent called, and what it returned
- Memory - a value the agent saved
- Hand-off - the conversation moving from one agent to another
- Knowledge - a search of your knowledge base
- Guardrail - a guardrail check, and anything it blocked
This is the agent's behaviour on the day the test was captured - the behaviour the test's checks were chosen from.
A test saved without a preview - an older one, or one you imported - shows the conversation only. A scenario generated by AI shows nothing until its first run. To see what the agent did in any run, open that run from Recent runs and expand Trace events on the test.
Tip: For the full guide to what a trace contains and how to read it, see Reading an execution trace. For a step-by-step playbook when a test fails, see Debug a failing test case.
Edit a test and its checks
Click Edit to change:
- Details - the test's name and category.
- Conversation - edit any turn or add new ones. The whole conversation is replaced when you save.
- Checks - under Assertions for turn on each turn, click Add assertion, or remove one that no longer proves anything.
Each check has a type and a Priority. A new check starts as Normal.
| Check | What it proves | Worth knowing |
|---|---|---|
| Tool called | The agent called a tool on the turn you add it to, so timing counts | Match by Name only for tools that take personal details; choose Subset of args or Exact args only when the details sent are what you're testing |
| Memory set | The agent saved a value | Check that it Exists, Equals a value, or contains one |
| Max turns | The conversation finished within a set number of turns | A suggested limit matches the turns the preview took, which is usually loose. Tighten it only when length matters |
| Priority | When the check fails |
|---|---|
| Critical | The test fails, and the evaluators don't score it |
| Normal | The test fails; the evaluators still score it |
| Informational | It's reported, but the test doesn't fail |
If you delete a turn that a check pointed at, the check moves to Unplaced assertions. It still runs and fails every time, so remove it or add it again on a turn.
A scenario generated by AI has no conversation or checks until it has run once. Run it, then come back to review and edit what it recorded.
Run a single test
Run test runs just this test. It's the fastest way to check a fix: change the agent, run the one test that failed, and read the result in Recent runs before you run the whole dataset again.
Delete a test
Delete removes the test for good, and you can't restore it. If you only want it out of one dataset, remove it from that dataset under Datasets instead.
When the agent changes
A test's latest result describes the agent as it was at that run. After you change the agent's instructions, tools, routing or knowledge, run its tests again - all of them, not just the one you were fixing.
If a test now fails, decide which side is wrong before you edit anything:
- The agent got worse. Leave the test alone and fix the agent. A test that keeps failing is doing its job.
- The requirement changed. Update the expected outcome, the conversation or the checks to match, then run the test again.
- The test was never right - it expects something nobody wants, or contradicts the agent's own rules. Fix the test, or delete it.
Good practice
- Compare the captured conversation with the failing run. The Conversation tab shows what the agent did when the test was captured; Trace events in the failing run show what it did this time. The first turn where they differ is usually where it broke.
- Add checks as you learn what breaks. Start with the one check that proves the main thing happened. Add more when a failure shows you something the test missed.
- Make the checks that must hold Critical. If the test means nothing without it, the evaluators shouldn't get a say.
- Don't edit a test to make it pass. If the test was right and the agent is wrong, the test should keep failing until the agent is fixed.
Next: Reports.