Barge-in That Actually Works: Tuning VAD for Phone Audio

One detector, two completely different jobs

Voice activity detection sounds like a single thing, and in most codebases it is a single component. But it is asked to answer two questions that have almost nothing to do with each other, and tuning it as one setting is where the trouble starts.

The first question is: has the caller finished their turn, so may I answer now? That is endpointing, it runs while your agent is quiet, and being wrong costs you either a long awkward gap or a sentence chopped in half. The second question is: is the caller talking right now while I am talking, so should I stop? That is barge-in, it runs while your agent is speaking, and being wrong costs you either an agent that steamrolls people or one that falls silent every time a lorry goes past.

Why studio settings fall apart on a phone line

Nearly every published benchmark for a voice activity detector is run on clean, wideband, single-speaker audio recorded through a decent microphone. A phone call is none of those things. Traditional telephony gives you eight kilohertz narrowband, which throws away most of the frequency range that makes consonants distinguishable, then compresses what remains.

On top of that, the caller channel is rarely quiet even when the caller is. There is comfort noise injected deliberately so the line does not sound dead. There is a television, a car, an open-plan office, a toddler. There is hold music from a system the caller is also on. A detector tuned to be helpful in a studio will read a good deal of that as speech, and each false positive either interrupts your agent or convinces it the caller is still talking when they are waiting for an answer.

The false barge-in, and what to do after one

The single most useful control in this whole area is a minimum interruption duration: a rule that says a burst of detected speech on the caller channel does not count as an interruption unless it lasts long enough to be a word. A cough, a door, a syllable of somebody else's conversation all fail that test and your agent keeps talking.

agent speaking:  "Your appointment is confirmed for Tues-"
  [vad]   speech detected on caller channel   0.12s
  [turn]  below minimum interruption duration (0.30s)
  [vad]   caller channel silent again          0.19s
  ---> not an interruption. agent keeps speaking.

agent speaking:  "Your appointment is confirmed for Tues-"
  [vad]   speech detected on caller channel   0.34s
  [turn]  interruption committed, agent stops
  [stt]   caller said: "sorry, Wednesday"
  ---> real interruption. agent yields.

On our own pipeline that threshold sits at 300 milliseconds, tuned down from the framework default of 500. Shorter than the default because on a real phone line a genuinely impatient caller starts speaking decisively, and half a second of waiting before yielding is long enough for two people to be talking at once, which is the exact situation barge-in exists to prevent.

The more interesting question is what happens after a false positive slips through anyway, because some always will. Naive implementations stop the agent mid-sentence, wait for a transcript that never arrives because the noise was not speech, and leave both parties in silence until the caller says something. The caller has no idea the agent was ever going to finish that sentence.

[turn]  interruption committed, agent stops mid-sentence
  [stt]   ...no transcript. line noise, not speech.
  [turn]  false interruption timeout reached  (1.5s)
  ---> agent resumes its own sentence instead of standing there

The fix is a timeout with a resume: if an interruption produced no transcript within a second and a half, treat it as false and let the agent continue its own sentence. That turns a dead call into a barely noticeable stumble, and it is the difference between an agent that seems fragile and one that seems composed.

The dial you are actually turning

Once you separate the two jobs, the settings stop looking like a config file and start looking like a product decision. Every one of them trades the same two failure modes against each other, and there is no combination that eliminates both.

The reason this matters more than it looks is that the endpointing floor is a hard lower bound on how fast your agent can possibly reply. Set it high to fight false interruptions and you have quietly capped the responsiveness of the entire product, in a way that no model upgrade will ever recover. We took that argument apart in the latency budget of a phone call, where turn detection turns out to be the largest single block in a one second gap.

Who owns the dial: pipeline or provider

There is a structural fork here that catches teams out when they switch architectures. In a classic recognition, model and synthesis pipeline, turn taking is yours: the detector runs on your side and you own every threshold. In a native-audio realtime model, turn taking belongs to the provider, and you get whatever controls they chose to expose.

On the realtime side we expose the provider's turn-detection parameters per assistant rather than hiding them, with the ranges clamped so a typo cannot produce a call that never yields. An out-of-range value logs a warning and keeps the default, because a bad config should degrade to sane behaviour instead of breaking a live call.

llm_config.turn_detection = {
  "threshold":           0.5,    // 0.0 - 1.0    how much signal counts as speech
  "prefix_padding_ms":   300,    // 0 - 2000     audio kept from before the onset
  "silence_duration_ms": 200     // 100 - 2000   silence before the turn is closed
}

Two of those parameters are deliberately not exposed at all, and the reason is worth stating plainly: the calls that create and interrupt a model response are what the conversation loop is built out of. Letting a customer disable either one produces an assistant that cannot hold a conversation, which is a support ticket rather than a feature.

Defaults also differ by model in ways that are easy to miss. Where a text-adjacent realtime model may close a turn after two hundred milliseconds of silence, we default a native-audio model to a considerably longer window and the least twitchy sensitivity setting available, precisely so that a background utterance during a pause does not open a turn. Porting one provider's numbers to another is a reliable way to make a working agent feel broken.

Why we took the cloud turn detector out of the call path

There is a fashionable option in this space: hand turn detection to a hosted service that uses a smarter model to predict whether a sentence is semantically finished, not just acoustically quiet. It genuinely works better on ambiguous pauses. We ran it, and in July 2026 we deliberately moved back to running detection locally.

Three reasons, in order of how much they mattered. First, it put a network call from a third-party cloud service directly in the live conversation path, which is the one place in a telephony system where an extra dependency is least welcome. Second, it carried a request quota that was comfortable at our volume and would not survive the scale we are building for, which makes it a ceiling rather than a component. Third, and the one that decided it: when that service was slow to respond, the framework fell back to plain local detection without saying so, which meant our turn-taking behaviour quietly changed depending on someone else's response time.

That is the same reasoning we apply to failover and to configuration generally: a system that degrades loudly is safer than one that degrades quietly. It is also why the local detector settings above are platform-wide rather than per assistant. They describe how our agents listen to a telephone, and that should not vary per customer without a conversation about why.