Reading an execution trace
A trace is the step-by-step record of what your agent did in each turn of a test: the tools it called, what it passed them and what came back, the values it saved, which agent took the conversation, and how long each turn took. The conversation shows you what the agent said; the trace shows you what it did to get there.
Open it when a reply is wrong and you need to know where it went wrong. Did the agent look the order up at all? With the order number the customer gave, or one it made up? Did it save the email the next step needs? Did the right agent pick up the conversation? The trace answers each of these with a yes or a no, so you fix the step that broke instead of guessing.
It's also what your checks are verified against. "The agent looked up the booking on turn 2" passes or fails on what the trace recorded. Evaluator scores are different: an AI judge reads the conversation and scores how good the replies were. The trace tells you what happened; the scores tell you how well. See Get started for how the two fit together.
Where to find it
| Where | How you get there | Use it when |
|---|---|---|
| A run's report | Open the run from Run history, expand a test, and open Trace events. | You want to know why a test passed or failed in that run. |
| A test's conversation | Open the test in the Testing Lab and go to Conversation. | You want to see what the agent did when the test was first captured. |
| Run preview | Shown as the preview plays out, while you create a test for a specific agent. | You're deciding which suggested checks to keep. |
The Conversation tab shows the run Nexus captured when the test was created with Run preview - a test written for a specific agent, or one created from Conversations. Imported tests aren't previewed, so their Conversation tab shows the messages only. To see what the agent does with them, run the test and read the trace in the run's report.
While you're trying ideas in the Playground, you can also look at what the agent did for each reply there.
What a trace shows
A trace is grouped by turn - one turn per agent reply. Each turn starts with a summary line, followed by one row per step in the order it happened. Read them top to bottom and you can follow how the agent got to its reply.
The turn summary
| You see | What it means |
|---|---|
| Turn 2 | Which agent reply this is. |
| OK, Error or Aborted | The turn finished normally, hit an error, or was cut short. An Error turn also shows the error code of the step that failed, such as a timeout. |
| A time in milliseconds | How long the agent took to reply. |
| LLM calls and tool calls | How many times the agent called the model, and its tools, in that turn. |
| mocked | How many tool results in that turn came from saved mock responses instead of your real systems. |
The summary is where you spot a turn that got the right answer the slow way - three model calls and two tool calls where one of each would do.
The steps
| Row | What it tells you | The question it answers |
|---|---|---|
| Tool call | Which tool the agent called. Show args lists exactly what it passed. | Did it look the order up with the number the customer typed, or one it made up? |
| Result | What came back: ok and how long it took, or failed with an error code. Show result shows the data returned. | Did the lookup fail, or return nothing that the agent then misread? |
| Memory | A value the agent saved, and whether it was added, changed or removed. When a turn changes several values, expand the row to see each one's old and new value. | Did the agent keep the customer's email so a later step can use it? |
| Hand-off | Which agent took the conversation, and which agent it came from. | Why did billing pick this up instead of support? |
| Knowledge | What the agent searched your knowledge base for, and the documents and passages it got back. | Did it find the refund policy, or something off-topic? |
| Guardrail | A guardrail that checked the turn, and whether it blocked, redacted or flagged it. | Was the reply blocked by a guardrail, or did the agent choose not to answer? |
A turn that shows No agent actions recorded for this turn. is one where the agent replied without calling a tool, saving a value or handing over. That's often correct; it's a problem when the reply needed a lookup it never did.
How checks use the trace
Every check you keep on a test is verified against the trace, which is why a check either passes or fails - there's no partial credit:
| Check | Passes when the trace shows |
|---|---|
| Tool called | The tool was called - on the turn you chose, if you chose one. |
| Memory set | The value was saved. |
| Max turns | The conversation finished within that many turns. |
In a run's report the checks are listed under Assertions, failures first. A failed check says what happened instead (Observed) and, when it's about a particular turn, which one - so you can go straight to that turn in Trace events.
When a Critical check fails, the test has already failed, so the evaluators don't score it and you'll see Evaluator scores were skipped. That's expected: the failed check and the trace tell you what went wrong. How Normal and Informational checks count is covered in Reports.
Things that look wrong but aren't
<redacted>- passwords, tokens and keys are hidden in the trace. The step still ran with the real value; the trace just doesn't show it. If you need to confirm the value, check where it comes from, not the trace.- truncated - very large arguments or results are cut short so the trace stays readable. A cut-off result isn't an empty one: check before you decide the tool failed.
- mocked and canned - the result came from a saved mock response, so it shows canned instead of a time. On that step the agent never reached your real system, which is what you want when you run with Enable Mocks on.
The trace and the scores
A test can fail a check while scoring well on every evaluator: the agent wrote a fluent reply but skipped the lookup it was supposed to do. It can also pass every check and score badly: the right tool ran, but the reply was unhelpful or off-tone. When a test fails, settle what happened in the trace first, then read the scores for how good the replies were.
Tips
- Open the trace before you guess. It answers most "why did it do that?" questions outright: a tool that never ran, a value that was never saved, an off-topic knowledge result.
- Stop at the first step that's wrong. Everything after it usually follows from it.
- Check the turn summary too. A correct but slow turn is easy to miss in the conversation and obvious in the summary.
- Don't take an empty-looking value at face value. Make sure it isn't hidden or cut short before you chase it.
Next: Debug a failing test case - turn what you read in the trace into a fix.