A native-audio voice model hears audio and answers in audio, with no text step in the middle. That single fact changes how you prompt it. The instincts you built on text models and text-to-speech engines quietly work against you here, and the result is an agent that sounds almost right and behaves almost right, which on a phone line is the same as wrong.

Three tools you keep confusing

A classic voice pipeline is three separate models in a row: speech recognition turns audio into text, a language model decides what to say, and a text-to-speech engine turns that text back into sound. You prompt the language model for content and you tune the speech engine with parameters like voice, stability, and speed. Two independent sets of controls.

A native-audio speech-to-speech model such as Gemini 3.1 Flash Live collapses all three into one. There is no separate speech engine to reach for. Everything you used to express as a text-model instruction and everything you used to dial in as a speech parameter now has to live in the same place: the system instruction, written in plain language.

The one-sentence version

In a pipeline the prompt is a brief for a writer and you handle the voice separately. In a speech-to-speech model the prompt is a stage direction for a voice actor, and it decides both what is said and how it sounds.

The speech knobs are now words

When there is no text-to-speech layer, there is no stability slider, no rate control, no separate voice config to lean on. Pace, register, warmth, how long an answer runs, whether the agent interrupts or waits: all of it is steered through the instruction itself. If you want the model to sound a certain way, you have to describe that way.

System instruction, style section

You are Sam, the receptionist for a dental practice.
Speak calmly and briefly. Keep answers to one or two
sentences. Never talk over the caller: if they start
speaking, stop immediately and listen. Warm, not chatty.

One rule from Google's own guidance is worth internalizing early: if you want a specific accent, name the output language alongside it, because accent and language are entangled inside a single audio model in a way they never were for a standalone speech engine.

The instruction is your only lever, and you get one shot

Here is the part that surprises people coming from text models. With a text language model you can keep feeding context mid-conversation: inject a fact, rewrite the system prompt, nudge the model between turns. On the realtime native-audio previews, several of those doors are closed. Google's Live API documentation states plainly that on the 3.1 model the calls used to regenerate a reply, update the instructions, and update the running context are not compatible. The voice framework we build on marks the same model's instruction and context as fixed for the life of the session.

What that means in practice

The system instruction you provide at the start of the call is the instruction you have for the whole call. You cannot quietly correct the model halfway through the way you would patch a text agent. Whatever behaviour you need has to be in the prompt before the first word is spoken.

This raises the stakes on the opening line too. The normal way to make a model speak first is refused on this model, so a scripted greeting cannot be a separate step. If you want the agent to open with an exact sentence, that sentence belongs pinned inside the instruction, not fired as its own command.

The rules that hold up, translated for voice

Google's best-practice guidance for the Live API is short and mostly about structure. Define the persona concretely: name, role, character. Put your conversational rules in the order you expect them to run, and separate the things that happen once (greet, collect a name) from the loop that repeats (the ongoing conversation). Describe a tool call in its own distinct sentences rather than one long compound clause.

Language handling gets an explicit recipe. If the agent must answer in a non-English language, say so twice, in the affirmative, and keep the model's language setting aligned with the caller's spoken language.

Language handling, verbatim from the docs

RESPOND IN {OUTPUT_LANGUAGE}.
YOU MUST RESPOND UNMISTAKABLY IN {OUTPUT_LANGUAGE}.

The guidance is just as clear about restraint. Do not have the agent repeat back what the caller just said. Do not write a multi-page prompt: chain shorter prompts instead. And when a caller interrupts, discard the audio you were about to play immediately, so the agent does not keep talking over a person who has already started.

What quietly does not work

Three habits fail silently, which is what makes them dangerous. Nothing errors, the call just goes slightly wrong and you find out later.

The first is the invisible mid-call hint. On a text model you can slip context into the conversation without the user noticing. On this model the only channel that still works mid-session treats your text as user input, so the model answers it out loud. There is no such thing as a silent context update here. Plan for it up front instead.

The second is reaching for the expressivity features. Native-audio models advertise settings meant to shape emotion and spontaneity, but on the realtime previews we have found they behave unreliably enough that we steer tone through the wording of the instruction instead. Plain description is more predictable than a flag that the preview may not honour.

The third is the wall of text. A native-audio model handed a sprawling multi-page prompt tends to drift and stall more, not less, and since you cannot correct it mid-call, a bloated instruction is a bet you cannot walk back. Short and ordered beats long and thorough.

The trap in one line

Everything you cannot fix mid-call has to be right in the opening instruction. A speech-to-speech agent gives you no second draft during the conversation.

The mental switch

The reason borrowed intuition fails is that a pipeline invites iteration and a speech-to-speech model does not. With a writer plus a separate voice you can adjust either one, mid-flight if you have to. With a single audio model you are directing a voice actor you get to brief exactly once, at the top of the scene. So write the persona, pin the greeting, describe the sound in words, set the language twice, and keep it short enough that the model can actually hold it.

Prompting a voice that talks back is not prompting a language model and it is not configuring a speech engine. It is a third craft, and the sooner you stop treating it like the other two, the sooner your agent stops sounding almost right.

Sources: Gemini Live API best practices and Live API capabilities. The behavioural notes on the 3.1 model reflect our own engineering experience running native-audio agents on live phone traffic.

Gemini Live 3.1 Prompting Guide (Voice Agents)