Agent rules and trace in a run report
When a test targets an agent, the run is checked against that agent's own instructions, one rule at a time. The report shows which rules the agent followed and which it broke, with the line of the conversation that proves it. So you can see why a test passed or failed against what the agent was told, not only against the scenario - and you don't need to copy an agent's rules into a test's expected outcome to have them checked.
Where to find it
- Write or generate a test for an agent, then run it.
- Open the run in Run history and expand the test.
- Agent rules sits under the evaluator scores.
Tests that don't target a specific agent don't show it.
Read the summary
The top line counts the rules: how many were followed, broken, or didn't apply. If some couldn't be counted, the number is shown in amber.
Next to it is a Strictness score worked out from these rules, and a note that says whether it counted:
- Used as this run's Strictness - the score from the agent's rules is the Strictness score for this test.
- For review only - the Strictness evaluator's own score stands, and is shown alongside for comparison.
If none of the agent's rules came up in the conversation, you'll see No rule applied to this conversation.
Read each rule
Broken rules come first, then the ones that weren't counted, then the ones the agent followed. Each rule shows:
- the rule, as it's written in the agent's instructions
- the step it belongs to, and When: the situation it applies to, if the instructions give one
- the line of the conversation that shows it was kept or broken
- a short reason
| Status | What it means | Counted? |
|---|---|---|
| Followed | The conversation shows the agent kept the rule | Yes |
| Broken | The conversation shows the agent broke it | Yes |
| Didn't apply | The conversation never reached the situation the rule covers | No |
| Unverified | The line given as proof can't be found in the conversation, so the result can't be trusted | No |
| Not checked | No result came back for this rule | No |
Rules that didn't apply are folded away at the bottom; click to show them. On an agent with long instructions, most rules won't come up in any one conversation, so a long folded list is normal.
What to do with a broken rule
- Read the quoted line first. Make sure it really is a break, and not a rule that's worded loosely enough to read either way.
- If the agent broke it, fix the agent - its instructions, or the tool it relies on - then run all of the agent's tests again, not just this one.
- If the rule itself is wrong or out of date, change the instruction. The next run checks the agent against its new instructions. Past reports keep the rules they were checked against.
- If a rule never applies in any test, nothing is testing it. Write a test that reaches that situation, ideally one where the customer pushes against the rule. See What to test.
Trace events
Below Agent rules, expand Trace events to see what the agent did on each turn: the tools it called and what they returned, values it saved, hand-offs to other agents, knowledge searches and guardrail checks. Read a broken rule next to the trace for the same turn - it often shows why the agent did what it did, such as a tool that returned nothing. See Reading an execution trace.
Next: Evaluators & Rules.