The demo was perfect. Laptop, good microphone, quiet room, the agent answering in what felt like real time — the kind of thing that gets a project funded in about four minutes. Then we put it on a phone number, and it fell apart. Not because the model got dumber. Because a phone call is a fundamentally different physical medium than a laptop microphone, and every assumption baked into that demo — clean 48 kHz audio, sub-50 ms network, a user who politely waits for the agent to finish — was wrong on the PSTN. The transcripts still looked fine. The calls were unusable.
That gap between "the demo works" and "the phone call works" is the entire subject of this article. Voice agents are the rare system where the model is the easy part and the plumbing decides whether you ship. I've built enough agent systems — see agent harness design for the text-shaped version of this problem — to say with some confidence that voice is where architecture stops being a slide and starts being a stopwatch.
What is a voice agent, architecturally?
A voice agent is a real-time loop that turns speech into an action and back into speech, under a latency budget tight enough that the human on the other end doesn't notice the machinery. That last clause is what makes it hard. Everything else — the transcription, the reasoning, the synthesis — is solved technology you can buy. The budget is not.
There are exactly two ways to build the loop in 2026, and the industry has largely settled on which one to use when.
The cascaded pipeline chains three independent models: speech-to-text (STT) transcribes the caller, a text LLM decides what to say and what tools to call, and text-to-speech (TTS) speaks the answer. Three vendors, three failure modes, three bills. The speech-to-speech (S2S) architecture collapses all three into one multimodal model that ingests raw audio and emits raw audio, never round-tripping through text at all. OpenAI's GPT-Realtime line, Google's Gemini Flash Live, and Amazon's Nova Sonic are the ones you'll actually be choosing between.
graph LR A["Caller audio
(8 kHz on PSTN)"] --> B["VAD +
turn detection"] B --> C["STT
streaming partials"] C --> D["LLM
+ tool calls"] D --> E["TTS
streaming synthesis"] E --> F["Agent audio
back to caller"] D -.->|"function call"| G[("CRM / booking
/ knowledge base")] G -.->|"result"| D B -.->|"caller starts talking"| H["Barge-in:
kill playback"] H -.-> E
The cascaded pipeline. The solid path is the happy case; the dotted paths are where the engineering actually lives. Note that turn detection sits before STT and also feeds barge-in — one component deciding both "has the caller finished?" and "should I stop talking?" is the source of most voice agent bugs.
Why does the cascade still win in production?
Because S2S is better at the thing that demos well and worse at the things that get you to production. The cascade dominates production deployments in 2026, and the reasons are unglamorous:
- Telephony physics. PSTN audio is 8 kHz, narrowband, and often transcoded through codecs designed in the 1970s. S2S models earn their advantage by preserving prosody, emotion, and acoustic nuance — precisely the signal the phone network throws away. On a phone call you pay for S2S and receive much less of what you paid for.
- Tool calling. Voice agents that do anything useful — check an order, book a slot, look up a policy — need deterministic function calling. Text LLMs have years of hardening here. S2S tool support is real now but noticeably thinner.
- Auditability. The cascade produces a transcript as a natural by-product. In a regulated deployment, "we have the text of every call because the architecture generates it" is a much easier conversation than bolting transcription onto an audio-only model for compliance.
- Swappability. Three components means you can upgrade the STT model without touching the voice, or move the LLM behind a gateway with routing and fallback when a provider has a bad afternoon. With S2S, the model is the pipeline. When it degrades, you have no seams to work with.
S2S wins when naturalness is the product and latency is the differentiator: a consumer companion, a language tutor, anything where the caller is on a good connection and being interrupted mid-sentence gracefully matters more than calling six APIs. That's a real category. It's just not most enterprise work.
The latency budget is the architecture
The number that matters is time-to-first-audio: the gap between the caller finishing their sentence and the first sound coming back. Under roughly 300 ms feels human. Between 300 and 600 ms feels sluggish but survivable. Past 600 ms, people start talking over the agent or reaching for the keypad — you've lost them, and no amount of prompt quality wins them back.
Here's the honest part: published industry medians across millions of real calls sit around 1.4–1.7 seconds, with p99 out at 3–5 seconds. Almost everyone is shipping something that feels sluggish. If your pilot is at 1.5 s you're not failing, you're average — which is exactly why the teams that get under a second stand out immediately.
Budget it component by component, because "it feels slow" is not a debuggable statement:
| Stage | Typical cost | What actually drives it |
|---|---|---|
| Network / telephony ingress | 50–150 ms | Codec, carrier hops, geography. Fixed cost — pick a region near your callers. |
| Turn detection (endpointing) | 150–700 ms | The biggest and most controllable slice. Naive VAD silence timers live here. |
| STT finalization | 100–300 ms | Time to promote partial hypotheses into a final transcript. |
| LLM time-to-first-token | 200–600 ms | Prompt size, cache hit rate, provider load. Grows with conversation history. |
| Tool call (if any) | 100–2000 ms | Your own backend. Usually the worst offender, and usually ignored. |
| TTS time-to-first-byte | 80–300 ms | Model class. Realtime-tuned voices are dramatically faster than flagship ones. |
Add those naively and you get two seconds and a bad product. Which brings us to the one idea that makes the cascade viable at all.
Streaming overlap: the trick that makes three models feel like one
The stages must overlap, not queue. A pipelined voice agent sends partial transcripts to the LLM while the caller is still speaking, streams LLM tokens into TTS as they generate, and starts synthesizing audio from the first complete sentence rather than the last. Done properly this cuts perceived latency by roughly three to five times versus running the stages in sequence.
The mental model I keep coming back to: the pipeline should be doing work at every instant the caller is making sound. If any stage is idle waiting for a complete input from the stage before it, that idle time is coming directly out of your budget and going straight into the caller's perception of "this thing is slow."
sequenceDiagram participant C as Caller participant S as STT participant L as LLM participant T as TTS C->>S: audio frames (continuous) S-->>L: partial transcript Note over L: speculative prefill
on partials C->>S: "...for next Tuesday" S-->>L: final transcript L-->>T: first tokens Note over T: synthesis starts on
sentence 1, not the whole reply T-->>C: first audio out L-->>T: remaining tokens T-->>C: rest of audio
Why overlap beats queueing. The LLM has already prefilled context from partial transcripts before the caller finishes, and TTS starts speaking sentence one while the model is still writing sentence three. The caller's perceived wait is the gap between their last word and the first audio — not the sum of the stages.
This is also why "just use a faster LLM" is usually the wrong optimization. Time-to-first-token matters enormously; total generation time barely matters at all, because TTS is consuming tokens as fast as the model produces them and the caller is hearing sentence one while sentence four is still being written. A model that starts fast and writes slowly beats a model that thinks for 400 ms and then dumps the whole answer at once.
The trap: speculative prefill on partial transcripts is a correctness risk, not just an optimization. If you start the LLM on "cancel my" and the caller finishes with "...cancel my cancellation, actually keep it" — you have a model that has already committed reasoning tokens to the opposite intent. Fine for prefill (you discard and re-run). Genuinely dangerous if you let a tool call fire off a partial. Gate every side-effecting function call behind a finalized transcript. I have seen this book a real appointment from a sentence the caller never finished.
What does a voice agent actually cost?
Roughly $0.08–$0.15 per conversation-minute for a mainstream cascaded stack, plus about $0.013/min per telephony leg. That's the number to put in the business case. The interesting part is the split, because it's not where most people guess.
For a typical short call on a common production stack, the cost lands around: TTS 48%, STT 28%, LLM 23%, hosting under 1%. Speech synthesis — the part everyone treats as a commodity checkbox at the end of the pipeline — is usually the single largest line item. Premium realtime voices run around $35–$50 per million characters; flagship ultra-realistic ones hit $100/M and up. Choosing a voice is a budget decision disguised as a design decision.
But run the same math on a 30-minute call and the LLM share climbs to roughly 47%, because the entire conversation history gets re-sent to the model on every single turn. Your cost per turn grows linearly with conversation length even though the caller's question is the same size. Long calls have a quadratic cost curve hiding inside them, and nobody notices until support conversations that were supposed to last four minutes start lasting twenty.
Two mitigations, both worth designing in from day one rather than retrofitting: aggressive prompt caching and a real memory strategy so history isn't re-billed verbatim on every turn, and a summarization checkpoint that compacts old turns once a conversation crosses some threshold. The second one is why agent memory types stop being an academic taxonomy and start being a line on your invoice.
Cascade or speech-to-speech: how I'd actually decide
| If you need… | Choose | Why |
|---|---|---|
| Phone / PSTN as the main channel | Cascade | 8 kHz audio throws away the acoustic nuance S2S charges you for. |
| Reliable multi-step tool calling | Cascade | Text LLM function calling is years more hardened. |
| Full transcript for compliance | Cascade | You get it free; with S2S it's an extra system. |
| Per-component vendor control or data residency | Cascade | Three seams to negotiate instead of one take-it-or-leave-it model. |
| Minimum possible latency, WebRTC channel | S2S | One model, one hop, no inter-stage handoffs. |
| Emotional nuance, prosody, natural interruption | S2S | It never destroys the acoustic signal by flattening it to text. |
My default for enterprise work is the cascade, and I don't think that's a close call today. But I'd build it behind an abstraction — which is exactly what frameworks like LiveKit and Pipecat give you — so that the day S2S tool calling gets genuinely good, swapping the middle of your pipeline is a project and not a rewrite. I've scored those frameworks properly against a full rubric in the voice agent platform assessment on the Architecture Radar, including the parts of the build/buy decision that don't fit in an article.
What to carry away
Voice is the domain where the model is the commodity and the pipeline is the product. The cascade wins most production deployments in 2026 not because it's more elegant but because it's more separable — you can debug it, audit it, swap pieces of it, and price each piece independently, and on a phone call you're not sacrificing much to get that. Speech-to-speech is genuinely better at being a conversation partner and genuinely worse at being a component in a system you have to operate.
Budget your latency stage by stage before you write a line of code, because "it feels slow" isn't debuggable and the sum of six innocent-looking numbers is a product nobody wants to use. Make the stages overlap; a pipeline that ever waits for a complete input is spending the caller's patience on nothing. And model your cost on a thirty-minute call, not a three-minute one, because the shape of the bill changes completely as the transcript grows.
The thing I'd tell anyone starting: the hard problem isn't getting the agent to say the right words. It's getting it to say them at the right moment — knowing when the caller is done, and knowing when to stop talking because they aren't. That's turn-taking, and it deserves its own article.