A realtime voice agent listens to an audio stream, determines when the user has finished a turn, reasons or calls tools, and produces speech quickly enough to feel conversational. Unlike text chat, it must handle silence, background noise, overlapping speech, interruptions, partial transcripts, and a user who cannot inspect a long answer before hearing it.
Voice can use a cascaded pipeline such as speech-to-text, model, and text-to-speech, or a native realtime model that consumes and emits audio. The right architecture depends on controllability, latency, language support, transcription needs, tool behavior, and privacy requirements.
The agent still needs the same tool, permission, evaluation, and observability controls as a text agent. Audio is an interface, not an exemption from system design.
| Approach | Strength | Tradeoff |
|---|---|---|
| Cascaded STT -> model -> TTS | Inspectable transcripts and replaceable components | More pipeline latency and error propagation |
| Native realtime audio model | Natural timing and low-latency interaction | Less component-level control and evolving APIs |
| Push-to-talk | Clear turn boundary and predictable control | Less natural for open conversation |
Voice activity detection can tell whether speech is present, but a pause does not always mean a thought is complete. Semantic turn detection combines acoustic and language signals to reduce cases where the agent cuts the user off or waits too long.
Endpointing delay balances responsiveness against interruption. Tune it with real conversations across languages, accents, noisy rooms, phone audio, and users who pause while thinking. A single global threshold rarely serves every channel equally.
Push-to-talk or explicit end-of-turn controls are often better for high-stakes tasks because they make intent clear. Natural conversation is not always the safest interface goal.
When a user interrupts, stop queued audio quickly so the agent does not continue talking over them. The conversation history should include only the portion the user actually heard, otherwise the model may assume it communicated facts or questions that never reached the user.
Short acknowledgments such as “okay” or background speech may be false interruptions. Track whether the interruption contained meaningful words and support resuming speech when the stop was accidental.
A user interruption should not cancel an already authorized backend write unless the workflow defines that behavior. Separate speech playback state from tool execution state and tell the user when an action is already committed.
Users experience one delay, but engineers need to see its parts: network transport, audio buffering, speech recognition, turn detection, model time to first token, tool calls, speech synthesis, and playback. Instrument each phase with timestamps and percentiles.
Streaming can begin a response sooner, but do not speak an unverified claim or irreversible decision simply to reduce latency. Use short acknowledgments while a tool runs, and make those acknowledgments honest about what is still pending.
Preemptive generation may start before the turn is fully confirmed. It can improve responsiveness but wastes tokens when predictions are discarded and can amplify an incorrect partial transcript. Apply it only where evaluation shows a real benefit.
Names, dates, addresses, confirmation codes, and amounts are easy to mishear. Read critical values back in a compact form and ask for confirmation before using them in consequential tools. Where possible, show the values in a visual companion interface.
While a tool runs, use a cancellable status such as “I am checking the order now.” Do not say the order is cancelled until the tool returns a verified success result. For long operations, offer a callback or asynchronous notification rather than filling time with invented progress.
Tool errors need spoken recovery that is brief and actionable. Preserve the detailed code and trace for operators, but tell the user whether to repeat information, wait, choose another path, or transfer to a human.
Tell users they are interacting with an automated agent. If audio or transcripts are recorded, explain the purpose and retention under the laws and policies that apply to the deployment. Provide a way to decline recording or use another channel where required.
Voice resemblance is not reliable authentication. Do not authorize account access, money movement, or sensitive disclosure because a caller sounds like a known person. Use trusted identity checks and step-up authentication outside the model.
Protect audio, transcripts, synthesized speech, tool results, and call metadata as separate data classes. Redact traces, minimize retention, restrict reviewer access, and avoid sending unnecessary personal information to providers.
A transcript alone misses whether the agent interrupted the user, spoke stale audio, paused awkwardly, or completed a tool before confirmation. Store a privacy-controlled event timeline with speech starts and stops, turn decisions, partial transcripts, tool calls, approvals, model output, playback, and handoff.
Test varied microphones, packet loss, noise, crosstalk, accents, code-switching, silence, emotional speech, corrections, and attempts to interrupt during a tool call. Use human listening review alongside automated metrics because naturalness and clarity are experienced through audio.
Model a voice turn as timed events: audio arrival, speech detection, partial transcription or audio understanding, end-of-turn decision, model response, tool request, approval, tool result, speech generation, playback, interruption, and repair. A transcript alone cannot show whether the agent talked over the user or announced success before a tool completed.
Keep playback state separate from tool state. When a user interrupts, stop unheard audio and align conversation history with what actually played, but do not pretend an already committed external action was cancelled. Tell the user whether the operation is pending, completed, failed, or requires follow-up.
Budget time to first audible response and time to useful completion by phase. Use short acknowledgements only when truthful, parallelize safe independent preparation, and preserve confirmation for names, amounts, destinations, permissions, and irreversible actions. Test silence, noise, accents, crosstalk, packet loss, correction, and human transfer with synchronized audio and trace review.
An interruption stops playback, but a committed tool action has its own lifecycle.
from dataclasses import dataclass
@dataclass
class VoiceTurn:
playback: str = "speaking"
tool_state: str = "not_started"
heard_words: int = 0
def interrupt(turn: VoiceTurn, heard_words: int) -> None:
turn.playback = "stopped"
turn.heard_words = heard_words
def commit_tool(turn: VoiceTurn) -> None:
turn.tool_state = "committed"
turn = VoiceTurn()
commit_tool(turn)
interrupt(turn, heard_words=7)
print("Playback:", turn.playback)
print("Tool:", turn.tool_state)
print("History keeps heard words:", turn.heard_words)Playback: stopped
Tool: committed
History keeps heard words: 7The application can now tell the user that speech stopped while the already committed operation continues.
A per-phase ledger shows which part of the turn consumes the response budget.
latency_ms = {
"audio_buffer": 80,
"turn_detection": 180,
"model_first_token": 420,
"tool": 240,
"speech_first_audio": 160,
}
budget_ms = 1200
total = sum(latency_ms.values())
largest = max(latency_ms, key=latency_ms.get)
print("Total:", total, "ms")
print("Within budget:", total <= budget_ms)
print("Largest phase:", largest)Total: 1080 ms
Within budget: True
Largest phase: model_first_tokenOptimizing the largest phase is more useful than making every component slightly faster.
0 of 2 checked
Try this next
0 of 3 completed
No. Native audio can improve naturalness and latency, while a cascaded pipeline can provide clearer transcripts, component choice, and control. Evaluate both against the product's languages, tools, privacy needs, and failure modes.
There is no universal number. Measure time to first audible response and time to useful answer, then test whether the interaction feels responsive without cutting users off or skipping verification.
Explore 500+ free tutorials across 20+ languages and frameworks.