Voice - Best Practices
Practical guidance from teams shipping Nexus voice bots. Read once before launch, revisit before any voice-related change.
Flip through them as flashcards - try to answer each question before you reveal it - or switch to Read as article for the full text.
Pick the right runtime mode
| If your goal is… | Pick |
|---|---|
| Production voice bot, broad provider/language support, full custom voice access | Text & TTS (pipeline) |
| Lowest possible latency for a short demo, with a small voice catalog | Realtime Audio |
| Anything multilingual or with a custom-cloned voice | Text & TTS (pipeline) - Realtime can't use your custom voices |
Most production deployments stay on Text & TTS for the lifetime of the bot. Don't switch modes unless you have a specific reason.
Pick the right TTS provider
| Provider | Sweet spot |
|---|---|
| Yellow AI (default) | Broad coverage, full custom-voice support. Right default for most bots. |
| ElevenLabs | Best-in-class English voice quality. Premium voices for high-stakes brand interactions. |
| MiniMax | Multilingual presets including Mandarin. Good for Asia-Pacific deployments. |
Don't switch providers per agent within the same bot if you can avoid it - keeps the brand voice consistent.
Pick the right STT (speech recognition)
Yellow AI is the default and right for most cases. Switch only when:
- Your audience speaks a language Yellow handles less well - try Deepgram, Sarvam (Indic), or Microsoft Azure.
- You're getting consistently bad transcripts despite good audio - try a different provider as a diagnostic.
Test STT with real recordings of your audience, not just your own voice. Accents, background noise, and domain vocabulary all matter.
Cloning a voice
- Record clean. A 5-second clean clip beats a 30-second noisy one. Quiet room, good mic, single take.
- Read the suggested sample text. It's tuned for clean clones.
- Name for the voice character, not the use case. "Aria - warm support EN" beats "Customer Support Voice."
- Test cloned voices in the Voice Playground and on a real phone before assigning to a production bot.
VAD tuning
Default values (threshold 0.85, prefix 300 ms, silence 500 ms) are right for most calls. Don't tune speculatively.
When you do tune:
- Slow / non-native speakers → raise silence duration to 700-800 ms.
- Fast-paced sales / outbound → drop silence duration to 300 ms.
- Noisy environments → raise threshold to 0.9.
Tune one knob at a time, retest, and document why you changed it.
Voice instructions vs bot identity
- Bot identity says what the agent does and is.
- Voice instructions say how it speaks - pace, language, tone delivery.
Don't repeat identity in voice instructions. Layer, don't duplicate.
Latency
Conversational voice feels broken above ~2 seconds end-to-end (user finishes speaking → user hears reply start). Aim for under 1.5 seconds.
Where latency hides:
- Slow model - large prompts, slow model choice. Trim the system prompt and try a faster model.
- Slow KB lookup - too many results, too-aggressive history concatenation. Tune the KB tool.
- Slow TTS - provider choice, voice choice. Some voices are slower to start than others.
- VAD too patient - silence duration high. Drop it.
- Telephony round-trip - fixed cost; can't tune from the bot config.
Measure before you optimize. Isolate model latency from TTS latency by swapping each in turn and re-testing in the Voice → Telephony Web Call.
Tool-call acknowledgements (_voice_ack)
The other big latency hide is the silent gap on tool calls. When the model dispatches a tool call (KB lookup, workflow, API call), the tool may take a second or two to return. Without anything to play, TTS goes silent - and to a voice user that silence reads as "the bot is broken."
Nexus voice handles this automatically. On voice channels, the runtime decorates every tool's parameter schema with a _voice_ack field - a short spoken phrase the LLM is expected to fill in whenever it dispatches a tool ("Let me check that for you", "One moment while I pull that up"). The runtime emits the _voice_ack value to TTS immediately, then strips it from the tool's actual arguments so the tool implementation never sees it.
You don't configure _voice_ack - it's automatic on voice channels. But:
- If the model skips the field (uncommon, but happens), the runtime falls back to your bot's static
toolCallFiller. Make sure you've set a sensible fallback ("One moment.") in the voice config. - The acknowledgement is one short utterance per tool call, not a sentence per argument. If a tool routinely takes more than 3 seconds, follow up the ack with a second message inside the tool's response stream rather than relying on the ack alone to cover the wait.
- Realtime Audio mode (OpenAI Realtime) handles this differently - see Voice on Nexus for the realtime-mode notes.
Multilingual voice bots
- Set the spoken language explicitly in voice instructions. "Respond in Hindi" is more reliable than expecting language detection.
- Cloned voices generalize across languages, but always test in the target language before going live.
- STT accuracy varies dramatically by language. Pick the STT provider that's best for your audience's language(s) - not just the default.
Escalation paths
A voice bot without an escalation path is a customer service liability.
- Add a Transfer Call tool. Wire it to a human destination.
- Add a Routing Logic rule that forces escalation on explicit triggers ("agent", "human", "representative", or repeated failure).
- Customize the escalation announcement so the user knows what's happening.
See Escalation tools.
Recording compliance
Recording rules vary by jurisdiction. Before enabling recording:
- Confirm the legal posture for each region you operate in.
- Add a clear consent prompt at the start of the call when required.
- Configure the Recording action on Transfer Call tools according to policy.
When in doubt, talk to legal before launch.
Pre-launch checklist
- Voice mode set deliberately (Text & TTS for production, Realtime only for narrow use cases).
- Provider and voice tested on a real call, not just the playground.
- Cloned voices (if any) tested in target languages.
- VAD tuned for the audience's speaking pace and environment.
- Voice instructions cover language and pace explicitly.
- Telephony provider configured for inbound and outbound.
- Caller ID set sensibly.
- Escalation tool wired and tested end-to-end.
- Fallback wired to a human-handoff workflow (not a generic apology string).
- Recording compliance reviewed for each region.
- Voice regression suite (5+ real test prompts) defined and run before each release.
Common mistakes
- Switching voice providers casually. Each provider has different voice character and latency profile. Customers notice.
- Skipping outbound testing. "It sounds great in WebRTC" doesn't mean it sounds great over PSTN.
- Tuning VAD without measurement. Random changes usually make things worse.
- No escalation path. A frustrated voice user without a human option churns immediately.
- Long welcomes. Voice users tolerate even shorter intros than chat users. Get to the point.
- Multilingual without locking the language. Don't trust auto-detection in production. State the language in voice instructions.
- Not listening to real calls. Production recordings (where compliant) are the only ground truth. Schedule time to listen.
Return to: Voice Overview.