Skip to main content

Create test cases

A test case is one conversation plus what you expect to be true about it. You author cases in the Testing Lab - click New test case to open the drawer, which has two tabs.

New test case drawer on the Import Content tab - four content-type cards (Chat Transcript, Email, CSV File, Generate with AI) and an Import from conversation form with Session ID and User ID inputs and an AI Enrichment toggle
1/2Import content: bring in a chat, email, or CSV

Two routes​

Import content​

You bring an existing conversation in as the test case - usually faster than typing it from scratch. Pick a content type:

  • Chat Transcript - import an existing conversation by Session ID + User ID from any past session.
  • Email - paste an email message or thread.
  • CSV file - upload a CSV with one or more conversations to bulk-import.
  • Generate with AI - seed from an agentic flow (similar to Scenario, but the LLM uses the agent's own flow definition as the prompt).

With AI Enrichment on (default), the platform auto-generates the test case name and expected outcome after import. Edit any field before saving.

Scenario (AI-drafted)​

You describe what the case should check; the LLM drafts the conversation.

  1. Give a title hint (optional), the agent's goal, any conversation rules to apply, and the test inputs you want covered.
  2. Click Generate with AI. The LLM drafts:
    • A short user ↔ agent conversation (typically 2-6 turns).
    • 1-3 expected behaviours.
  3. Edit any field before saving - the draft is a starting point, not a contract.

💡 Try the Nexus AI: "Draft 10 adversarial test prompts for my billing agent - jailbreaks, off-topic, hostile users, edge cases that exploit ambiguous policy."

💡 Try the Nexus AI: "Generate a Scenario test case where the user asks about returns mid-flow during a refund request."

Trial run review​

On a Nexus bot with a known target agent, composing a case (from either tab) doesn't save it straight away - it first routes through a Trial run preview. The platform runs the case against the live agent, streams each turn back as it goes (user message, agent reply, trace events), and captures the result as an immutable baseline (see Baseline capture). For imported chat logs and CSVs it uses replay mode - the captured user turns are replayed verbatim instead of being re-paraphrased, which roughly halves the wait.

Once the preview finishes, the Assertion picker opens so you can choose which suggested assertions ship onto the case, then click Approve to save.

Legacy-platform bots and agentless drafts skip this step and save directly - no baseline, no suggested assertions.

Assertion picker​

When a case routes through the Trial run preview, the backend proposes a set of suggestedAssertions - checks that would have passed against the trace it just captured. Examples:

  • "getOrderStatus tool was called on turn 2"
  • "Final response contained the tracking number"
  • "Bot did not invoke the transferCall tool"

The picker lets you tick which suggestions ship onto the saved case:

  • Turn-anchored suggestions (tool_called with turnIndex) render inline beside the matching agent message, with the same card chrome as the trace events panel.
  • Non-turn-anchored suggestions live in a panel below the conversation.
  • The picker pre-filters out internal $$ memory paths and trivial max_turns suggestions - you can still pick them if you want, but they're noise by default.

Only the assertions you check are written to the case. Everything else is treated as informational signal, not a hard pass/fail criterion. The bulk-approve paths apply the same filter automatically.

Baseline capture​

When a case routes through the Trial run preview, the backend stores the full execution trace as a baseline (a SimulationReport flagged isBaseline). The baseline gives the Test Case detail page something to pair each agent message against (tool calls, memory updates, per-turn metrics).

Baselines are immutable - re-running a case produces new trace data, but the original baseline stays so you can always diff against the moment the case was captured.

Run status semantics​

A case is either ready (inputs unchanged since the baseline - safe to run) or stale (inputs the case references - memory keys, user-profile fields, variables - were renamed, retyped, or removed in the agent config after the baseline was captured; the case will still run but the result may not be comparable to the original baseline).

The Testing Lab table surfaces a stale chip on affected rows, and the same signal appears as a banner on the Test Case detail panel. Either re-capture the baseline (Trial run with the new inputs) or accept the staleness if the change was intentional.

Read next: Run tests - queue a dataset and read the results.

Last updated on