The Latency Budget of a Phone Call

Two different latencies, constantly confused

Ask three people how fast a voice agent is and you will get answers to three different questions. The first is answer latency: how long between the phone being picked up and the agent saying its opening line. The second is turn latency: how long between the caller finishing a sentence and hearing a reply. The third is what most benchmarks actually measure, which is time to first token from a language model sitting on a GPU with the audio already transcribed.

Only the second one is what people mean when they say a voice agent feels slow. Answer latency is a one-off at the start of the call and callers forgive it, because a ring followed by a short pause is what a phone has always done. Time to first token is a real number, but on a phone call it is one term out of eight, and optimising it in isolation is how teams end up shaving fifty milliseconds off a gap that is over a second long.

The chain, stage by stage

Here is what actually happens between a caller falling silent and hearing a voice again, on a normal inbound call over a SIP trunk with a classic recognition, model and synthesis pipeline. Read it as a budget rather than a benchmark: the proportions are the lesson, and every one of these stages is a place a millisecond can hide.

The figures for recognition, model and synthesis are the order of magnitude providers publish for their own components, and they will move depending on which ones you pick. The first and last stages are network. The second one is the interesting one, and it is the reason this article exists: before a single byte reaches your speech recogniser, your system has already spent longer deciding that the caller was finished than the model will spend answering.

caller stops speaking            t+0
  [rtp]  last voiced packet arrives     t+38ms    (jitter buffer)
  [vad]  silence run starts             t+38ms
  [vad]  silence threshold reached      t+438ms   <-- nothing has been asked yet
  [turn] endpointing confirms turn end  t+438ms   (floor reached, dynamic)
  [stt]  final transcript emitted       t+571ms
  [llm]  first token                    t+864ms
  [tts]  first audio byte               t+1002ms
  [rtp]  first packet leaves            t+1041ms

  caller-perceived gap                  1.04s

The 400 milliseconds in front of your model

A voice agent cannot answer until it believes the caller has stopped speaking, and silence is not a signal that arrives. It is the absence of one, which means the only way to detect it is to wait and see whether it continues. Every millisecond of that wait is dead air on the line, and it is spent before the pipeline has been asked to do anything at all.

This is a genuine trade-off rather than a bug to be fixed. Cut the wait short and the agent interrupts people mid-thought, treating the pause before a phone number as the end of a sentence. Stretch it out and the conversation acquires a lag that makes every exchange feel like a satellite link. There is no setting that makes both problems go away, only a position on a dial that suits the calls you actually take.

On our own pipeline the wait is bounded rather than fixed. Turn detection runs in dynamic mode with a floor of 400 milliseconds and a ceiling of 2 seconds, so a confident end of turn is committed quickly while an ambiguous one gets room to finish. Underneath it, voice activity detection needs 400 milliseconds of continuous silence before it will call a speech run over, with an activation threshold of 0.4 and a minimum speech duration of 50 milliseconds so that a cough does not open a turn.

The mechanics of that dial, and how to keep it from cutting people off, are their own subject. We wrote them up separately in barge-in that actually works, which covers why settings tuned on clean studio audio fall apart on an 8 kHz phone line.

What you cannot optimise

Some of the budget is not yours. Traditional telephony carries audio as 8 kHz narrowband, packetised into 20 millisecond frames, and every hop that touches it adds a little. The jitter buffer is the honest one: it deliberately holds packets so that uneven arrival times can be smoothed into steady audio. Shrink it and you trade delay for gaps and clicks. It is latency you are choosing to buy, and buying it is correct.

If you are transcoding audio between codecs because two ends never agreed, you are paying a hop on every packet for the length of every call. That one is worth fixing before you change a single model.

The only real trick: stop waiting in series

Read the budget again and the structural problem is obvious. Each stage waits politely for the one before it to finish. Silence detection completes, then recognition finalises, then the model starts reading, then synthesis starts speaking. Added up in series, a chain of individually reasonable numbers produces an unreasonable gap.

The fix is not to make the stages faster. It is to stop running them one after another. Recognition can stream partial transcripts while the caller is still talking. Synthesis can start speaking the first clause while the model is still writing the last one. And the model can be asked to start drafting a reply before the turn is formally confirmed, on the bet that the caller really has finished, which is what preemptive generation does on our pipeline. When the bet is right the reply is already warm. When it is wrong the draft is thrown away and nobody hears anything.

1. 01

   Stream partial transcripts instead of waiting for a final

   A recogniser that emits as it goes lets everything downstream start earlier. Waiting for a final transcript is the single most common way teams accidentally serialise the pipeline.
2. 02

   Start synthesis on the first clause, not the last full stop

   The caller only needs to hear the beginning of the sentence on time. Buffering a complete answer before speaking any of it converts model latency directly into silence.
3. 03

   Generate preemptively during the endpointing wait

   The wait is dead time by definition. Spending it on a draft that may be discarded costs tokens and buys back the most expensive block in the budget.
4. 04

   Say something while a tool call is running

   If answering requires a lookup, the honest fix is not a faster lookup but a holding phrase, so the silence is filled by the agent instead of by the caller wondering whether the line dropped.

Speech-to-speech changes the shape, not just the size

A native-audio model hears audio and answers in audio with no text step in between, which collapses recognition, reasoning and synthesis into one hop. That removes real stages from the budget rather than shortening them, and it is why realtime models feel different rather than merely faster.

What comes with it is a different set of constraints, not a free win. Turn detection moves to the provider, so the dial you were tuning is now theirs and exposed through whatever knobs they choose to give you. Several things you could do mid-call in a pipeline stop being available. And the instruction you open with is, on some models, the only instruction you get for the whole call. We covered that trade in the prompting guide for native-audio voice models.