Debug a failing test case

A failed test tells you something is wrong; this page helps you find what, and fix the right thing. The short version: open the failed test, find the first step in its trace that went wrong, and fix that. Most failures come down to a single step - the wrong tool ran, a tool failed, a value was never saved, the conversation went to the wrong agent.

Work the trace before the scores. The trace tells you what happened; a low evaluator score only tells you a reply was poor, not why.

The debugging loop

1. Open the run. In the Testing Lab, open Run history and click the run. Filter the list to the failed tests.
2. Expand a failed test. Its checks come first, under Assertions, with failures at the top. Each failed check says what happened instead (Observed) and, if it's about a particular turn, which one. Below them are the evaluator scores, the agent rules the agent followed or broke, and Trace events.
3. Find the turn that went wrong. Open Trace events, go to the turn the failed check names (or the first reply that looks wrong), and read its steps in order until you reach the one that's wrong.
4. Decide which side is wrong - the agent or the test. See Is it the agent or the test?
5. Fix it, then run the test again. Open the test in the Testing Lab and click Run test, or use Retry failed on the report to run just the failures again. Read the new run's trace to confirm the step now does what it should.

Tip - run one test while you iterate. It's quicker than running a whole dataset. Once the fix holds, run all of the agent's tests: a change that fixes one conversation can break another.

Symptom → where to look

Map what went wrong to the row in the trace that holds the answer:

Symptom	Where to look	Likely fix
The wrong tool ran, or one ran that shouldn't have	The Tool call rows for that turn - the tool's name.	Two tools with overlapping descriptions, or instructions that point at the wrong one. Make the descriptions distinct, or say in the instructions when to use each.
The right tool, with the wrong details	Show args on the Tool call row.	The agent misread or invented a value. Tell it where the value comes from, or have it use one it already saved.
A tool failed	The Result row says failed, with an error code; the turn shows Error.	A timeout or an error from your own system isn't the agent's fault - fix the system, or turn on Enable Mocks for the run. If the agent sent bad details, fix those as above.
The tool worked, but the answer was wrong	Show result on the Result row.	If the data was right, the agent misread it - tighten its instructions. If it was empty, check the tool, and whether the test's user profile has the data it needs.
A value the agent needed later wasn't saved	The Memory rows on the turn it should have been saved.	See A value wasn't saved.
The wrong agent took over, or none did	The Hand-off row - which agent it went to, and from.	See It went to the wrong agent.
A knowledge answer was wrong or empty	The Knowledge rows - what was searched for, and what came back.	Nothing useful came back: the content is missing or unclear in your knowledge base. The right passage came back and the agent ignored it: fix the instructions.
The reply was blocked or changed	A Guardrail row marked blocked, redacted or flagged.	A guardrail stepped in, so this usually isn't the model's doing. If it shouldn't have, review your AI safety & conduct settings.
The turn was slow	The turn summary - its time, LLM calls and tool calls.	A slow tool, or more model and tool calls than the job needs.

Note: Passwords, tokens and keys show as <redacted>, and very large values are cut short and marked truncated. Neither means the value was wrong or missing - the trace just doesn't show all of it.

The most common failures

It called the wrong tool, or passed the wrong details

The most common failure: the agent answered, but used the wrong action, or the right one with bad inputs.

1. Find the Tool call rows for the turn that went wrong.
2. Check the tool's name. If the wrong tool ran, the agent couldn't tell your tools apart, or its instructions sent it to the wrong one. Make each tool's description say clearly when it applies, so no two overlap, or tell the agent when to use each.
3. Check the details it passed with Show args. A wrong order number, a missing field, or a value the customer never gave means the agent misread the conversation. Tell it where the value comes from. If the value was captured earlier, have the agent use the saved value rather than work it out again each turn - see Memory.
4. Run the test again and confirm the right tool now runs with the right details.

Tip - name or details? They look alike in the result but are fixed in different places. A wrong tool is a choice problem (descriptions and instructions); the right tool with wrong details is a reading problem (where the value comes from).

A value wasn't saved

A later turn behaves as if it never learned something - it asks for the order number again, or loses the customer's email.

1. Go back to the turn where the value should have been captured.
2. Look for a Memory row for that value:
   • It's there, with the right value - the save worked; the problem is later. Check that the later step reads the same variable that was saved - order_id and customer.order_id are different variables.
   • It's there, with the wrong value - whatever produced it captured the wrong thing. Treat it like a wrong-details problem above.
   • It isn't there - the save never happened. Check whether the tool or step that sets it ran, and whether its Result says failed.
3. Fix it and run the test again, and confirm the Memory row now shows the right value.

It went to the wrong agent

The conversation moved to an agent that shouldn't have handled it, or stayed put when it should have moved.

1. Find the Hand-off row in the turn that went wrong, and note which agent it went to and came from.
2. Compare that with your routing. Two agents whose Trigger descriptions overlap will be picked unpredictably. Make each one say clearly when that agent applies, or add a rule in Routing logic to settle it.
3. Run the test again and confirm the conversation goes where it should.

caution

If routing is right most of the time and only fails on one phrasing, don't broaden a rule to catch it - that tends to break other conversations. Keep this test, and make a narrow change.

Multi agents: the wrong path

In a multi agent, the usual symptom is a step that finished on the wrong exit, so the conversation went down the wrong branch.

1. Run the multi agent on its canvas and open the Logs tab to see which exit each step finished on. See Test a multi agent.
2. Compare it with the step's exits. A wrong exit almost always means two exit descriptions overlap - sharpen them so each describes a distinct outcome. See Exits.
3. If a step sends an extra message as it leaves and the next step talks over it, tell the step to finish without a further message, and turn on Exit without a reply.
4. If a step hit on_error or max_retry, make sure that exit leads somewhere sensible. An exit that leads nowhere leaves the conversation stuck.

Nexus Tip

Ask the Nexus AI to read a failed test and name the step that broke, with the change it suggests. You decide whether to make it.

Ask the AI layerThis test is failing on turn 3. Read its trace, tell me which step went wrong, and what to change to fix it

Is it the agent or the test?

Not every failure means the agent is broken. Before you change anything, decide which side is wrong:

What you find	What to do
The agent didn't do what it should	Fix the agent, then run all of its tests again.
The test expects something nobody wants - for example exact wording the agent was never told to use	Fix the test: edit it, and loosen or remove the check that's too strict.
The requirement has changed	Update the test first, then the agent.
The test shows Systematic error	The run couldn't finish the test, so there's no verdict on the agent. See below.

When a test errors instead of failing. Systematic error means the run couldn't finish the test - the report says at which stage, for example Agent response failed. When every test in a run errors - for example because a system the agent calls was down - that's not a verdict on the agent. Turn on Enable Mocks when you start the run, so the agent gets your saved mock responses instead of calling that system, and run it again.

When the results are Outdated. Results describe the agent as it was when the run happened. If you've edited the agent since, its Tests button shows them as Outdated - run the tests again before you act on them. See Test signals on the agent page.

When it fails only sometimes. If a test passes when you run it again with nothing changed, run it several times in one go with Iterations (available when you run 10 tests or fewer). If the result keeps flipping, either the agent wavers - tighten its instructions - or the test is unclear about what counts as right.

caution

Don't lower the bar to make a test pass. If the test was right and the agent is now wrong, the test should keep failing until you fix the agent.

Tips

• Stop at the first step that's wrong. It's usually the cause; everything after it follows from it.
• Fix one thing, then run again. Change five things at once and you won't know which one fixed it, or which one broke something else.
• Keep the conversation that exposed the bug. Once it's fixed, a real conversation that went wrong is worth keeping as a test - create one from it in Conversations, so it can't come back unnoticed.
• Don't debug from the scores alone. A low score tells you a reply was poor; the trace tells you which step to change.

---

Read next: Reading an execution trace - every row in a trace, and what it tells you.

Back to: Get started - what to test, how to read a run, and how to keep your tests trustworthy.