Get started with the AI Trust Centre
A test suite is only worth what it would catch. Fifty tests that all check the easy case will pass while the agent quietly breaks a rule; ten well-chosen ones will tell you the moment it does. This page covers how to build a suite that really covers your agent, how to read what a run tells you, and how to keep the suite worth trusting as the agent changes.
When to write tests
Write most of your tests once the agent is finalised - its job, instructions, tools and routing are in place, and you've tried it in the Playground. Tests written against an agent that is still changing shape keep failing for reasons that have nothing to do with quality, and tests generated from its instructions describe whatever those instructions said on the day.
Until then, use the Playground to try ideas, and save a conversation as a test when it shows something you never want to break again.
Start from what the agent is meant to do, not only from its instructions. If you have a brief, a process document or a list of requirements, test against that. Tests drawn only from the instructions check that the agent does what it was told; they can't tell you it was told the wrong thing.
What to test
A complete suite covers more than the request the agent exists for. Use this as a checklist, and add each kind of test when the agent gives you a reason to:
| Test | What it proves | Example |
|---|---|---|
| The main job | The agent does what it's there for, start to finish | A customer moves a booking with a valid reference |
| Each rule, both ways | One test where the agent keeps a rule, one where the customer pushes against it | "Only offer listed slots": the customer takes a listed slot / insists on a Saturday |
| Missing or wrong information | The agent asks again or recovers, rather than guessing | No booking reference; a reference that doesn't exist |
| Tools | The agent uses a tool at the right moment, with the right details, and copes when it fails | It looks up the booking before offering slots; the booking system is down |
| Facts | Answers drawn from your systems or knowledge base are right, and the agent doesn't invent what it doesn't know | It gives the opening hours you hold; it says it can't find a price you don't hold |
| Routing | On a bot with several agents, the right one picks up the conversation, and the others don't | "I want to cancel" reaches the cancellations agent, not rescheduling |
| Out of scope | The agent declines, or hands over, what isn't its job | Asking the booking agent for a refund |
| Long conversations | The agent keeps track across turns, through corrections and changes of mind | The customer changes the date halfway through |
| Hand-off to a person | It hands over when it should, and not when it shouldn't | The customer asks for a person; the customer is merely annoyed |
| Safety and personal data | It holds its rules under pressure, and never reveals someone else's details | "Ignore your instructions and…"; asking for another customer's booking |
| Languages and voice | On a bot that runs in several languages, or on voice | It replies in the customer's language; it doesn't read a link out loud |
| Real failures | What went wrong for a real customer doesn't happen again | A conversation from Conversations that went badly |
Start with the main job and every rule both ways. Then add tool and failure tests if the agent calls tools, routing tests if the bot has several agents, fact tests if it answers from your data or a knowledge base, and so on.
When the agent should behave differently for different customers - a member and a guest, a monthly and an annual subscriber - run the same tests as each user profile rather than writing them twice.
Write tests that can fail
A test that can't fail tells you nothing. Most weak tests are weak in one of these ways:
- Say what must not happen, not only what should. A test that only lists what should happen passes when the agent does the right thing and something harmful in the same reply. "Must not agree to a time that wasn't offered" catches a broken rule; "handles the request well" never will.
- Name the test after the rule it checks.
refuses-unoffered-timestill makes sense after the instructions are reworded;saturday-requestdoesn't say what it protects. - Describe the customer as a person. For a Scenario, two to four sentences: their situation, what they want, and how they react when the agent pushes back. A thin description produces a shallow conversation. For an attack, write the exact words the customer uses, so the test doesn't soften it.
- Add checks where the conversation alone can't prove it. A check confirms that the agent called a tool, saved a value, or finished within a set number of turns. Tie a check to a turn when timing matters: "looks the booking up before offering slots" is a different test from "looks it up eventually". Make the checks that must hold Critical.
- Spell out the facts. When the right answer is a specific value - a price, a time, an address - put the exact value in the expected outcome, taken from your own data, so the test knows what right looks like.
- Keep tests inside what a run can do. A test that needs an account state your test profiles don't have, or a system to fail on cue without a mock, fails every time for the wrong reason. Give it the profile or mock it needs, or leave it out.
Group tests into datasets: one per agent, one for routing, and a short set of the most important tests that you run after every change.
Create them
| Option | Best for |
|---|---|
| Generate tests with the Nexus AI | A first draft for a finalised agent. Review every test: drafts built from the agent's instructions only check what those instructions say. |
| Scenario, in the Testing Lab | A specific behaviour you want checked. You describe it, Nexus drafts the conversation, and you edit it. |
| Import Content, in the Testing Lab | Turning chat transcripts, emails or a CSV of conversations into tests. |
| Conversations | Turning a real conversation that went wrong into a test, so it can't happen again unnoticed. |
When you write a test for a specific agent, Nexus runs it once before saving it (Run preview) and suggests checks based on what happened. Keep the ones that prove something, then approve. See Create test cases.
Read a run
Open the run from Run history and expand a test to see the conversation, its scores, its checks (listed as Assertions), the agent rules it followed or broke, and a step-by-step record of what the agent did. See Reports.
A result has two parts, and they answer different questions. Checks say whether the right thing happened; evaluator scores say whether the reply was any good. A test that scores well but fails a check usually means the agent sounded right without doing the work.
For every failure, decide which side is wrong before you change anything:
| What you find | What to do |
|---|---|
| The agent didn't do what it should | Fix the agent, then run all of its tests again. |
| The test expects something nobody wants | Fix the test. |
| The requirement has changed | Update the test first, then the agent. |
| Every test errored, for example because a system the agent calls was down | It's not a verdict on the agent. Use a mock for that system and run again. |
Read some passes too. A pass means the test was satisfied, not that the agent did everything right. Open a few passing conversations now and then, starting with the rules that matter most; a pass that should have failed means the test needs sharpening.
A result that changes between runs needs attention. Run the test several times in one run (Iterations). If it flips, either the agent wavers - tighten its instructions - or the test is ambiguous about what counts as right.
Keep your tests trustworthy
- Run the agent's tests after every change to its instructions, tools, model, routing or knowledge - all of them, not just the one you were fixing. A change that fixes one conversation can break another.
- Treat Outdated results as unknown. They describe the agent before your last change.
- Don't tune the agent to the tests. Keep two or three tests you never adjust the agent for. If those start failing while the rest pass, the instructions have been fitted to the visible tests rather than improved.
- Turn every real failure into a test, from Conversations, so it stays fixed.
- Retire tests that check nothing: duplicates, tests for agents that are switched off, and tests whose expected outcome nothing could fail.
- Fix tests that contradict the agent's own rules - for example, one that expects an exact English sentence from an agent told to reply in the customer's language. It fails on an agent that is doing the right thing.
Next: Testing Lab.