Debug a failing test case

A test case fails. Now what? This page is a practical playbook for turning a red row in the Testing Lab or Reports into a concrete fix. The short version: open the case, read the trace, find the one event that went wrong, fix that. Almost every failure resolves to a single event in the execution trace - the wrong tool fired, a tool errored, a memory write never happened, the conversation handed off to the wrong agent. The trace shows you which.

Note: The trace is deterministic - a tool either was or wasn't called, an agent either did or didn't hand off. That is separate from the fuzzy evaluator scores. When you're debugging, work the trace first: it tells you what happened, not just how well it scored. Evaluator scores tell you a turn was bad; the trace tells you why.

The debugging loop

The loop is the same every time:

1. A case fails in a Testing Lab run or a Reports run - you see a red status and, for assertion-based failures, which assertion missed.
2. Open the failing case. Click the case row (in Reports, click into the per-case result) to open the Test Case detail panel.
3. Go to the Conversation tab. Each agent message is paired with the execution trace that produced it - the ordered list of events for that turn.
4. Find the turn that failed, expand its trace, and read the events in order until you hit the one that's wrong.
5. Fix the bot config (tool, routing rule, agent instructions, KB content, guardrail), then click Re-run on the case and watch the conversation + trace re-render. The single-case Re-run is the fastest loop in the Trust Centre - you don't have to re-run the whole dataset to check a fix.

Tip - Re-run the single case, not the dataset, while you iterate. It's faster and cheaper, and the trace re-renders inline so you can confirm the fix without leaving the panel.

Symptom → where to look

Read the events for the failing turn top to bottom. Map the symptom to the event that carries the answer:

Symptom	Where to look in the trace
Wrong tool fired (or a tool fired that shouldn't have)	The tool-call events for that turn. Check the tool name and the arguments it was given - a wrong name is a routing/instruction problem; right name, wrong arguments is a prompt/argument-extraction problem.
Tool failed / errored	The tool-result event. Check its status (success vs error), the error code/message, and the duration - a timeout shows up as a long duration plus an error, which is a different fix than a 4xx from bad arguments.
Expected memory wasn't set	Look for the memory-update event for the variable (dotted paths like order_id or customer.email), and check its before/after. If there's no memory-update event at all, the write never ran - trace back to the tool or step that was supposed to set it.
Agent switched unexpectedly (or didn't switch when it should)	The agent handoff / transition event - which agent it moved from/to, and the stated reason. If the reason looks wrong, the fix is usually in Routing Logic or in the target agent's Trigger.
KB answer wrong or empty	The knowledge-base retrieval event. Check the query that was sent and the returned chunks with their match scores - low scores or off-topic chunks point at a content/indexing gap; a good chunk that the model ignored points at an instruction problem.
Reply was blocked, redacted, or flagged	The guardrail check event - whether it allowed / blocked / redacted / flagged the turn, and which guardrail. A blocked reply with no useful output is almost always a guardrail firing, not a model failure.
An assertion failed	Correlate the assertion to the trace event it checks: tool_called → the tool-call events; no_tool_called → confirm the tool is absent; response_contains → the agent message text; goal_status → the conversation summary's final status. The assertion names the contract; the trace shows whether the run honoured it.
Latency / cost too high	The per-turn summary - latency (ms), tokens in/out, number of LLM calls and tool calls. A turn with many LLM calls or a slow tool result is where the time/cost went.

Note: Secret-looking values (tokens, passwords, API keys) appear in the trace as <redacted>, and very large values are truncated. That's expected - a <redacted> argument doesn't mean the value was wrong or missing, just that the trace hides it. If you need to verify a redacted value, check the source config or memory, not the trace.

Step-by-step: the most common cases

Case A - "It called the wrong tool" (or the wrong arguments)

This is the most common failure. The bot answered, but it used the wrong action - or the right action with bad inputs.

Step 1 - Open the failing turn's trace

Conversation tab → expand the trace under the agent message that's wrong. Find the tool-call events for that turn.

Step 2 - Check the tool name

If the wrong tool fired entirely, this is a selection problem. Either two tools have overlapping descriptions (the model couldn't tell them apart) or the agent's instructions point it at the wrong one. Tighten the tool descriptions so they're mutually exclusive, or sharpen the agent's instructions about when to use each.

Step 3 - Check the arguments

If the right tool fired but the result was wrong, read the arguments it was given. A wrong order_id, a missing required field, or a value the model invented means the model misread the conversation. Fix the agent prompt to extract the argument more reliably, and consider whether the value should have come from memory (see Memory) instead of being re-derived each turn.

Step 4 - Re-run and confirm

Click Re-run on the case. Watch the trace re-render - confirm the correct tool now fires with the correct arguments before you move on.

Case B - "Expected memory wasn't set"

A downstream turn behaved as if it never learned something from an earlier turn - re-asked for an order_id, used a stale account_tier, lost the customer's email.

Step 1 - Find where the write should have happened

Walk back through the turns to the point the value was supposed to be captured. Expand that turn's trace.

Step 2 - Look for the memory-update event

• It's there, with the right value → the write worked; the problem is downstream (a later turn read the wrong key, or the key was overwritten). Check the dotted path the later turn reads against the one that was written - a mismatch like order_id vs customer.order_id is a classic.
• It's there, with the wrong value → the tool or step that produced it captured the wrong thing. Treat it like Case A (wrong arguments / wrong tool result).
• It's missing entirely → the write never ran. The step or tool that sets it either didn't fire or errored - check its tool-result event for an error, then trace why it didn't run.

Step 3 - Fix and re-run

Correct the variable path, the capturing step, or the tool, then Re-run and confirm the memory-update event now appears with the right before/after.

Case C - "It handed off to the wrong agent"

The conversation jumped to an agent that shouldn't have handled the message - or stayed put when it should have routed away.

Step 1 - Find the transition event

In the failing turn's trace, locate the agent handoff / transition event. Note the from agent, the to agent, and the reason.

Step 2 - Read the reason against your routing

If the reason cites a trigger or rule that shouldn't have matched, the fix is in Routing Logic or the target agent's Trigger. Two agents whose triggers overlap will route ambiguously - tighten the triggers so they're mutually exclusive, or add a deterministic Routing Logic rule to break the tie.

caution

If the bot routes correctly most of the time and only fails on this one phrasing, don't broaden a rule globally - promote this exact conversation into a test case and add a narrow rule. Broadening a rule to fix one case often breaks three others. (See Routing Logic on keeping rules deterministic.)

Step 3 - Re-run and watch the transition

Re-run the case and confirm the transition event now moves to the right agent for the right reason.

Debugging guided agents

Guided agents fail differently: the symptom is usually a node took the wrong exit, so the flow branched to the wrong next node. The flow is deterministic by design, so the trace pins this down quickly.

1. Open the failing case and find the turn produced by the misbehaving node.
2. In the trace, read which exit the node finished on (the transition event shows the chosen exit and the next node it routed to).
3. Compare that against the node's configured exits. A wrong exit almost always means two exit descriptions overlap - the node matched the wrong one because they weren't mutually exclusive. Sharpen the exit descriptions on the Conversational Agent node so each describes a distinct, non-overlapping outcome.
4. If the node sent an extra chatty message before exiting (and the next node repeated it), that's an instruction problem on the node, not a routing one - tell the node to finish silently on the matching exit. See the node's best-practices in agent-node.
5. If the node hit on_error or max_retry, check that those exits are wired somewhere sensible - an unwired error path is a dead end that surfaces as a stuck or failed turn.

Nexus Tip

Ask the Nexus AI Layer to read the failing turn's trace and name the single event that broke - then propose the specific config change (tool description, routing rule, agent instruction, or KB content) and wait for your confirmation before applying it.

Ask the AI layerThis test case is failing on turn 3. Read the trace, tell me which event went wrong (tool, memory, handoff, KB, or guardrail), and what to change to fix it

When the fix isn't in the trace

A few failures aren't bot bugs at all:

• The case is stale. If the detail panel shows a stale-baseline banner, the bot config drifted after the baseline was captured (a variable was renamed, a rule was added). The case may be comparing against an outdated baseline - re-capture the baseline or accept the staleness. See Test Case.
• The assertion is too strict. Sometimes the bot is right and the assertion is wrong - a response_contains that demands exact wording the model phrased differently. Loosen or disable the assertion on the Assertions tab rather than "fixing" the bot to match a brittle check.
• It's a flake. If a case passes on Re-run with no config change, it may be non-deterministic. Run it a few iterations (the Run tests modal has an Iterations stepper) before treating it as a real failure.

caution

A stale chip is a warning, not a bug. If the case was right and the bot is now wrong, the case should keep failing - fix the bot, don't lower the bar by re-baselining a broken behaviour.

Best practices

• Read the trace top to bottom, stop at the first wrong event. The first event that diverges from what you expected is usually the root cause; everything after it is downstream noise.
• Fix one thing, then Re-run. Don't batch five config changes - you won't know which one fixed it (or which one broke something else).
• Promote the failure into a permanent case. Once you've fixed it, the conversation that exposed the bug is a test case worth keeping. Import it (see Testing Lab) and assert the fix so it can't regress.
• Distinguish "wrong tool name" from "wrong arguments." They look similar in the result but have different fixes - name is a selection/routing problem, arguments are a prompt/extraction problem.
• Don't debug from evaluator scores alone. A low score tells you a turn was bad; only the trace tells you which event to change.

---

Read next: Reading an execution trace - the full anatomy of a trace and every event type, so you know exactly what each line in the playbook above is pointing at.

Back to: Testing overview - the Playground, Trust Centre, and how the testing surfaces fit together.