The Rules in Your Prompt That Never Run

The question that decides everything: who owns the turn

Before you write a single line of a voice agent prompt, there is one thing worth knowing about the model underneath it: when the caller starts speaking while the agent is talking, who decides what happens next?

On most stacks, nobody asks the model. A voice activity detector sits in front of it, watching the caller's audio channel. The moment it decides that what it hears is speech, the platform stops the outgoing audio and tells the model about it afterwards. The model finds out that it was interrupted the way you find out your call dropped. It had no say, it has no memory of the sound that caused it, and no sentence in its prompt could have changed the outcome.

On a full-duplex model, the arrangement is different. The model hears the caller while it is speaking, in the same stream it is generating into, and it decides for itself whether to stop. That is a genuine judgement, and a judgement can be instructed.

The same logic applies at the other end. If there is a separate text-to-speech step, the words your language model writes are read out literally, so spelling is a sound decision. If the model produces audio directly, nothing reads your text, and a rule about how to write numbers has nothing to act on. We have written about where those milliseconds go in the latency budget of a phone call; this is the same architecture seen from the prompt side.

Four families, three different sets of levers

Sort the models you are likely to be offered by those two questions and they fall into four groups. What changes between them is not the tone of the prompt. It is which instructions are capable of having an effect.

The last row is worth a note. A model that hands its thinking to a second model needs the split written down, or you get an agent that either stalls while it waits or answers before it knows. We took that apart in what we had to change for a full-duplex model on a phone line, and the same shape shows up in how tool calls behave under a deadline.

On a pipeline, the spelling is the sound

A pipeline agent's language model writes text that a speech engine reads out loud, word for word. That makes the prompt partly a pronunciation document, and it is the one family where writing rules genuinely change what the caller hears.

Spell everything out. No digits in running text, no symbols, no abbreviations. Read the digits of a phone number or a reference one at a time, separated by commas. Where a numeral collides with another word in your language, write it so it cannot be misread. And be careful asking the model to convert anything: a clock time translated into words is one of the more common ways to be confidently an hour wrong, and a wrong time is worse than an ugly one.

Punctuation is a weaker tool than it looks. Commas work. Doubling them buys nothing. Ellipses and trailing dots do lengthen the pause, but they also introduce audible artefacts, because punctuation the speech model does not recognise is a risk rather than a rest. If you want real space between two subjects, write a short ordinary sentence with a full stop in it.

Never let yourself be interrupted. If the caller starts
speaking while you are mid-sentence, finish your sentence
first, then listen.

That instruction is well written and completely inert. Whether the agent yields is decided before the language model is involved, by a minimum interruption duration and a set of thresholds tuned on real call audio. If your pipeline agent cuts people off or steamrolls them, no rewording of the prompt will help; the fix is in how the detector is tuned for phone audio.

What does work here is everything about wording: what the agent says while it waits for a lookup, how it hands a caller over, how it phrases bad news. A handover in particular is worth writing out rather than leaving to the model, for reasons we went into in what happens when the transfer fails.

Two more instructions in this family tend to be dead on arrival. Asking for listening noises while the caller talks, because a pipeline is not listening while it speaks. And respelling brand names phonetically inside the prompt, which only affects the sentences the model happens to copy; pronunciation belongs in the replacement list, where it applies to every sentence the agent will ever say.

On speech-to-speech, the spelling is nothing

Take the same prompt to a native-audio model and the writing rules stop meaning anything. There is no text handed to a speech engine, so there is no spelling to get right. What you lose along with it is every knob you used to have on the speech side: no stability slider, no rate control, no separate voice config. Everything you used to set as a parameter now has to be described in words, inside the instruction.

We wrote a longer piece on that shift in prompting speech-to-speech voice models. The short version: the prompt stops being a brief for a writer and becomes a stage direction for a voice actor. “Speak calmly, keep answers to one or two sentences” does the work that a comma cannot do here.

One consequence of that last point catches people out. On several native-audio models the instruction is fixed once the session starts, so there is no quiet mid-call correction the way there is with a text model. Whatever behaviour you need has to be in the prompt before the first word is spoken, which is a strong argument for a short, ordered prompt rather than a thorough one.

The one model that listens to you about listening

Full-duplex models are the exception that makes the rule visible. Because the model hears the caller while it speaks, and decides its own turns, the instructions that were decorative everywhere else suddenly do something. You can tell it whether to make brief acknowledgements while the caller talks. You can tell it to yield to the caller but not to a cough. You can tell it what to do about a second voice in the room.

That last one matters more than it sounds. The single most common complaint about phone agents in practice is not a bad answer; it is a caller on speakerphone with somebody else in the room, where every stray sentence knocks the agent off its turn. On a model with detection in front of it, that is unfixable from the prompt. On a full-duplex model it is a paragraph.

Your conversation partner is only the person on the call.
A television, people in the room, or noise from the street are not.
A background voice never interrupts you: keep talking, do not react
to it, and never answer a question from someone else.
If you genuinely cannot tell, do not guess. Ask briefly whether they
were talking to you or to someone else.

Worth knowing before you rely on it: these models tend to separate the caller from the room on content rather than on voice. A background sentence that happens to sound like an instruction can still get an answer. Which is why the last line of that block is the important one: asking is better than guessing, and a short “were you talking to me?” costs a second and saves a derailment.

Why “now and then” becomes “constantly”

Here is the failure we did not expect, and the reason we are more careful with wording on these models than on any other. A full-duplex model hears the caller continuously. Give it an instruction to acknowledge the caller now and then, and it will evaluate that instruction on every pause. “Now and then” is not a frequency to a model listening without interruption; it is a standing permission.

The result is an agent that hums and murmurs through everything the caller says, which reads as attentive in a transcript and sounds exhausting on a call. The same thing happens with permission to take the initiative: told it may occasionally jump in with something useful, it jumps in whenever something useful occurs to it, which on a phone call is constantly.

The shape is: default, reason, named condition, maximum, and what to do when unsure. Any “sometimes” left unbounded will be resolved by the model, and it will resolve it more generously than you meant.

How to find the dead rules in a prompt you already have

You do not need a framework for this. Read your prompt with one question in hand for each sentence: is this a decision the model gets to make? Three passes catch nearly everything.

Pass one, turn-taking. Find every sentence about interrupting, waiting, stopping, or noise. If the agent is not on a full-duplex model, none of them run. Delete them and move the intent into the platform settings, where it will actually take effect.

Pass two, pronunciation. Find every sentence about how to write numbers, dates, symbols or brand names. If the agent is on a native-audio model, none of them run either. On a pipeline they do — but the brand-name ones belong in the pronunciation replacements rather than the prompt, so they work on every sentence instead of the ones the model copies.

Pass three, length. Whatever is left, cut. This is worth doing on any model and it is close to mandatory on the native-audio ones, where the instruction is fixed for the call and a sprawling prompt drifts more than a short one.

What you are usually left with is shorter than what you started with and does more, because every line in it is a line the model can act on. The rest was never running. It was just sitting there, looking like diligence.