Voice Agent Turn-Taking: Barge-In, Endpointing, and Testing

"My postal code is K1A zero..." — and the agent starts talking. The caller stops, confused, starts over. The agent, now hearing speech while it's mid-sentence, stops itself. Two seconds of mutual silence follow, where each party is waiting for the other. Then both start at once.

I've listened to a lot of call recordings like that. Every one of them came from a system whose latency dashboard was green. Median time-to-first-audio, 900 ms — respectable. The problem wasn't that the agent was slow. It was that the agent was fast at the wrong moment, and no aggregate metric anywhere in the stack could see it.

In the previous article I argued that voice agent architecture is really latency budgeting, and that turn detection is the biggest controllable slice of that budget. This one is about that slice specifically, because it's where the actual product quality lives, and it's the part I see teams under-invest in most consistently.

What is endpointing, and why is it so hard?

Endpointing is the decision that the caller has finished their turn and it's now the agent's job to speak. It sounds trivial. Humans do it without thinking, at roughly 200 ms accuracy, in noisy rooms, across languages, while distracted.

The naive implementation is a silence timer: run voice activity detection (VAD) on the incoming audio, and when the energy stays below a threshold for N milliseconds, declare the turn over. Pick N=500 ms and your agent interrupts people constantly. Pick N=1500 ms and every exchange has a full second of dead air welded onto it. There is no value of N that works, and that's the point: silence duration is not the signal.

Watch where it breaks. All of these are pauses in the middle of a turn, and energy-based VAD cannot tell them apart from the end of one:

• Spelling. "My last name is S — H — I..." Every hyphen is a silence longer than your threshold.
• Numbers read aloud. Card numbers, account numbers, phone numbers. People chunk them, and the chunk boundaries are pauses.
• Addresses. "Forty-two... Wellington Street... apartment three B."
• Thinking. "I need to change my... uh... the flight on the fourteenth."
• Dollar amounts. "Three thousand... two hundred and fifty."

Notice these aren't edge cases. They're the exact content of a customer service call — the sentences where getting it wrong costs the most. Your agent works beautifully for chitchat and falls apart the moment someone reads out an account number, which is to say it falls apart precisely when the call starts mattering.

Semantic turn detection: use the words, not just the silence

The fix is a model that predicts whether an utterance is complete, using linguistic and acoustic evidence rather than a stopwatch. "My postal code is K1A zero" is obviously unfinished — not because of how long the pause is, but because of what was said and how the pitch was still rising at the end. That's learnable.

Two open implementations are worth knowing, and they've made interestingly different bets:

Both run on CPU in the tens of milliseconds, which is the entire reason they're usable — a turn detector that costs 300 ms has eaten the savings it was meant to deliver. And both are strictly better than a silence timer on the cases listed above.

The nuance that matters: tune patience by workflow risk, not globally. One threshold for the whole call is a mistake. When the agent has just asked for an account number, it should become dramatically more patient — a long pause there is almost certainly mid-utterance. When it's just asked "anything else?", a short pause is a real answer and waiting a full second is insulting. The turn detector gives you a probability; what you do with it should depend on what you just asked.

graph TD
  A["Caller audio frame"] --> B{"VAD:
speech present?"}
  B -->|no| C["Accumulate silence"]
  B -->|yes| D{"Agent currently
speaking?"}
  D -->|no| E["Append to
caller utterance"]
  D -->|yes| F{"Barge-in classifier"}
  F -->|"backchannel
('mhm', 'yeah')"| G["Ignore,
keep speaking"]
  F -->|"noise / TV / crosstalk"| G
  F -->|"real interruption"| H["Stop playback,
flush TTS buffer"]
  H --> I["Truncate LLM context
to what was actually heard"]
  C --> J{"Semantic turn model:
utterance complete?"}
  E --> J
  J -->|"p < threshold"| K["Keep listening
(patience by workflow)"]
  J -->|"p >= threshold"| L["End turn,
run the agent"]

Barge-in is a policy, not a feature

Barge-in is the decision about what should happen when the caller makes sound while the agent is talking — and "stop talking" is only one of the available answers. A working implementation distinguishes at least four cases:

1. A real interruption. The caller wants the floor. Stop immediately — every 100 ms you keep talking is aggravating.
2. A backchannel. "Mhm," "right," "okay," "yeah." These are the listener signalling keep going. An agent that stops dead every time someone says "mhm" is exhausting to talk to, and this is the single most common barge-in bug I see.
3. Noise. A dog, a TV, a door, road noise, another conversation in an open-plan office. Ignore.
4. Crosstalk on the line. Echo of the agent's own audio coming back through a bad speakerphone. Ignore — and if you're not doing echo cancellation you'll see the agent interrupt itself, which is as funny as it is embarrassing.

Getting from "sound detected" to a correct classification needs more than an energy threshold: a voice classifier to reject non-speech, a minimum-duration guard so a click doesn't count, and ideally a quick semantic check on the first word or two. A pure energy gate will fire on all four.

Then there's the part that gets forgotten. When you cut off playback, the agent's context must be truncated to what the caller actually heard, not what you generated. If your LLM produced three sentences, TTS spoke one and a half, and the caller interrupted — the model's history must record one and a half sentences. Skip this and the agent will confidently refer back to information it never actually said out loud. The caller experiences an agent that hallucinates its own past.

What should you actually measure?

Latency percentiles are necessary and nowhere near sufficient. The four that catch what they miss:

These pull against each other in pairs, which is the useful part. Missed barge-in and false interruption are the two ends of one dial. P95 dead air and clipped words are the two ends of another. You are not optimizing four numbers; you're choosing two operating points, and you cannot know if you chose well by looking at either number alone. That's a tuning problem, and tuning problems need a test harness — the same argument I made about evaluating agent systems generally, with the added wrinkle that here the failure is inaudible in a transcript.

Which is worth stating plainly, because it reframes the whole testing strategy: a large share of production voice issues are voice-specific and simply do not appear in transcript-only evaluation. One widely-cited figure puts it around 42%. Read the transcript of the postal-code call at the top of this article and it looks fine — the words are all there, in order. You have to listen to it to know it was a disaster. Any eval suite that reads transcripts is blind to nearly half your failures by construction.

How do you test something this stateful?

Three layers, and the industry has converged on roughly this shape:

Golden conversations as regression tests. A fixed set of scripted calls with known-correct outcomes, replayed on every change. This is your smoke test — it catches the prompt edit that broke the booking flow. Cheap, fast, run it on every commit.

Simulation with adversarial personas. Hundreds to low thousands of synthetic callers per release, systematically varying accent, background noise, speaking rate, and interruption behaviour. This is where you find that the agent works for calm native speakers and collapses for someone calling from a car. Gate the build on category-level pass rates, not on one aggregate score — an overall 94% that's hiding 60% on accented speech is a fair-lending problem waiting to happen, not a passing grade.

Production replay. Real calls, re-run against the candidate build. Nothing you invent in a test lab matches the creativity of actual humans. Feed the failures back in as golden tests.

The habit I'd push hardest, and it costs almost nothing: after every change to the prompt, model, voice, or turn-detection settings, listen to the twenty most-interrupted calls. Not read — listen. Twenty calls is maybe forty minutes. It will tell you more about your agent's real quality than any dashboard, and it's how you catch the class of problem where all your numbers are green and the product is bad. Voice is the one domain where the eng team genuinely has to consume the output the way the customer does.

What to carry away

Turn-taking is not a detail of a voice agent — it's most of what people mean when they say the agent feels good or feels broken. The model choice, the prompt, the voice: all of it is downstream of whether the thing knows when to talk.

Silence duration is the wrong signal, and no threshold value fixes it, because the pauses that matter are inside utterances, not between them. Use a semantic turn detector, and vary its patience by what you just asked the caller to do. Treat barge-in as a four-way classification rather than a stop switch, and when you do stop, truncate the context to what was actually heard. Then measure the four metrics that pull against each other, because your latency dashboard will happily show green while you interrupt every caller who tries to read you their account number.

And listen to the calls. All the instrumentation in the world is a proxy for the thing you can just directly experience in forty minutes a week.