Evaluators & Rules
Evaluators decide how every reply in a run is judged: which qualities are scored, and how good a score has to be to pass. Set them once for your bot and leave them alone, so a run today can be compared with a run last week. Open Evaluators & Rules in the AI Trust Centre to change them.
An evaluator is an AI judge that reads the conversation and scores one quality of the agent's replies, such as accuracy or empathy. It answers was the reply any good? It can't tell you whether the agent actually did the work - whether it looked the booking up, or saved the reference. That's what the checks on a test are for. See Checks and scores.
The evaluators
There are ten, in two groups. Each one has a switch, and you can expand it to read what it measures.
Quality
Each Quality evaluator gives a score out of 100, and you set the threshold the score has to reach to pass.
| Evaluator | What it judges | Default threshold |
|---|---|---|
| Empathy | Whether the agent acknowledges the customer's concerns and how they feel | 70 |
| Accuracy / Quality Score | Whether it resolves the issue and gives correct information | 80 |
| Response Variability | Whether it varies its wording rather than repeating itself from turn to turn | 50 |
| Strictness | Whether it sticks to its instructions and guidelines | 80 |
| Hallucination | How much it makes up. Lower is better: a score above the threshold fails | 35 |
| Clear Communication | Whether its replies sound natural and human, without repetition | 70 |
| Follow-up Handling | Whether it handles follow-up questions and keeps track across turns | 70 |
Safety
Safety evaluators have no threshold. A reply either passes or fails.
| Evaluator | What it catches |
|---|---|
| Language Filter (Toxicity & Bias) | Replies that insult, mock, blame or demean the customer, or use biased or hateful language |
| PII Detector | Replies that leak personal details, such as email addresses, phone numbers or ID numbers |
| Jailbreak Evaluator | An agent that gives way when a customer tries to override its instructions |
Each evaluator passes or fails on its own. A test can pass Accuracy and fail Hallucination in the same run, and the report shows both. See Reports.
Tip - Strictness and agent rules. When a test targets a specific agent, the run is also checked against that agent's own instructions, one rule at a time, and the report says whether that result was used as the run's Strictness score. See Agent rules.
Change the scoring
- Open Evaluators & Rules and stay on the Evaluators tab.
- Switch off any evaluator that doesn't apply. Empathy means little for an agent that only checks an order status.
- Drag a threshold up to be stricter, or down to let more replies pass. Remember that Hallucination works the other way round.
- Click Save. Nothing changes until you do, and Cancel throws your changes away.
The new settings apply to every test on the bot from the next run onwards. Reports you already have keep the scores and thresholds they were judged with, so changing a threshold doesn't rewrite past results.
Reset to defaults puts every switch and threshold back to the recommended settings.
The Rules tab
The Rules tab lists the rules that come built in, grouped as Safety, Workflow and Tool. Search them, or pick a category to narrow the list. You can't add or change rules here.
To hold an agent to a rule of your own:
- Write the rule into the agent's instructions. Whenever a test targets that agent, the run checks it against every rule in its instructions, and shows which ones it followed or broke. See Agent rules.
- Add a check to the test, and make it Critical if it must hold - for example, that the agent looks the order up before it confirms a refund. See Edit a test and its checks.
- Write a test that pushes against the rule, not only one where the customer goes along with it. See What to test.
Good practice
- Set it once, then leave it. Runs are only comparable when they're scored the same way. If you change thresholds before every release, you can't tell whether the agent improved or the bar moved.
- Switch off rather than lower. If an evaluator doesn't matter for this bot, turn it off. Left on with a very low threshold, it only adds noise to every report.
- Use evaluators and checks together. Evaluators tell you how good the reply was; checks tell you whether the right thing happened. Most agents need both: "is polite to an upset customer" is an evaluator's job, "never confirms a refund without an order ID" is a check's.
- Reset to defaults when nobody remembers why. If several people have tuned the thresholds over months, the defaults are a clean baseline to start again from.
Next: Test Case.