GPT-Live-1 on a Phone Line: What We Had to Change

One model talks, another one works

A speech-to-speech model such as Gemini Live or Grok Voice hears audio, decides what to do, calls your tools and answers in audio, all in one model. Collapsing those stages is why realtime models feel different rather than merely faster, as the latency budget of a phone call explains. GPT-Live-1 keeps listening and speaking in one model, and it is full duplex: it listens while it speaks, and decides for itself when to take a turn and when to stop. What it deliberately leaves out is the work.

When a caller asks whether Thursday at ten is free, the voice model delegates that question to a backend text model. It keeps the caller company while the backend reasons, checks the calendar and returns a result, and then puts that result into its own words. OpenAI suggests pairing it with a fast model such as GPT-5.6 Luna for routine work like scheduling, and a larger one such as GPT-6 Astra for problems that need real reasoning.

There are two ways to run that backend. With Responses delegation, GPT-Live calls a model you configure through OpenAI's Responses API, and a framework such as LiveKit Agents runs your tool functions for it. With client delegation, your application receives the task, runs any model, agent or service you like, and decides what goes back to the voice. For most phone agents Responses delegation is the shorter road. Client delegation earns its complexity when you need to validate, redact or combine results before they are spoken. Either way, the backend is where the requests to your business systems happen, with every constraint described in tool calls to business systems and the per-system quirks on the integrations pages.

Two prompts instead of one

The split changes what a prompt is. The voice model gets a short prompt about the conversation: who the assistant is, how it sounds, how it handles interruptions and when to delegate. The backend model gets the business rules and the tool procedures. OpenAI's prompting guide recommends a delegation policy in the voice prompt with three parts: what the backend can do, when to delegate, and when not to.

You are Sam, the receptionist of a dental practice.
Speak calmly. One or two sentences per answer.
If the caller interrupts, stop and listen.

Delegation policy
Backend tools:
- Appointments: check free times, book, move or cancel.
Delegate to the backend when:
- The caller asks for a time, or wants to book, move or cancel.
- The caller corrects a request that is already in progress.
Do not delegate when:
- You can answer from the conversation so far.
- The caller greets you or asks you to repeat something.

Check availability with check_availability before offering a time.
Only book within opening hours: Monday to Friday, 08:00 to 17:00.
Confirm name and date of birth before changing an existing appointment.
If a tool fails, say so and offer a callback. Never guess an outcome.

Two traps sit on either side of this. Do not leave the backend prompt empty: the backend does not automatically receive the voice model's instructions, so without its own prompt it decides about tools without knowing the rules of the business. Do not move everything into the voice prompt either. A voice model carrying a page of business rules gets slower and stiffer, which is the same lesson as in prompting native-audio voice models. Logic about who is calling, such as recognising a caller from their number, belongs with the backend.

An interruption stops the voice, not the work

Full duplex makes interruptions feel natural, and that hides a trap. OpenAI's delegation guide is explicit that interrupting speech does not cancel backend work. If a caller says “actually, make it Friday” while a Thursday booking is running, the voice stops and adapts, and the backend may still book Thursday.

So a correction has to reach the backend as a correction. Your application decides whether to cancel the running task, change it, or let it finish and discard the result, and a cancellation request is not proof that anything was cancelled. Before retrying a tool that seemed to fail, check whether the action already happened, so a lost response does not turn into a second booking. The principle is the same as when a transfer fails: say what you know, and never paper over an outcome you are not sure of.

What behaves differently on a phone line

Most of the differences come from the model owning the conversation more completely than the realtime models before it.

The barge-in row matters more on a phone than in a browser. Narrowband 8 kHz audio, line noise and hold music all trip detectors that were tuned on clean audio, as barge-in that actually works covers in detail. And when a call goes wrong at the telephony layer rather than in the model, the SIP logs are still where the cause is written down.

Four things that fail quietly during setup

None of these raises an error you would notice in a demo. Each one produces a call that connects and then does not behave.

1. 01

   Pin the endpoint

   Without an explicit base_url, the OpenAI plugin takes its address from the OPENAI_BASE_URL environment variable. If anything on the machine sets that variable, for a local model server or a proxy, GPT-Live connects there and the caller hears silence. Pass https://api.openai.com/v1 explicitly.
2. 02

   Test every voice before you offer it

   A voice name that your framework accepts is not necessarily one OpenAI accepts. At launch, aster was in LiveKit's voice list but refused with forbidden: Voice session access denied, which leaves the agent silent. Start one session per voice before putting it in front of callers.
3. 03

   Change the backend model only after a tool call

   Start with the default backend and switch only after a test call in which a tool actually runs. A backend that is not delegating properly does not sound broken: the voice stays friendly, and nothing gets booked.
4. 04

   Upgrade the agent packages together

   GPT-Live support arrived in LiveKit Agents 1.8.1. If your stack pins an older release, move all the LiveKit agent packages to the same version rather than only the OpenAI plugin.

Two meters on the bill

The voice layer costs $0.05 per minute, billed per second, and it reports no tokens. The backend model is billed separately at its normal token prices, and so are the tools it uses. A minute of GPT-Live is therefore a fixed voice charge plus a variable amount that depends on how often the assistant delegates and which backend it delegates to.

That has two practical consequences. A cost dashboard that counts tokens per component will show zero for the voice part, which is correct and misleading, so track seconds for the voice and tokens for the backend. And an assistant that looks something up on nearly every turn costs noticeably more than one that mostly talks. The meters that run whichever model you choose are in what a voice agent really costs per minute, and how a platform fee stacks on top is on our pricing page.

Where it fits, and where it does not

It fits calls where the conversation itself is the hard part: callers who interrupt, hesitate, change their minds or talk over background noise. OpenAI reports a response latency of 0.798 seconds against 1.41 for GPT-Realtime-2.1. Treat that as the vendor's number and measure your own chain end to end, because on a phone call the model is only one of eight stages that add up.

It does not fit when a sentence has to be spoken exactly, such as a mandatory disclosure. A pipeline with text-to-speech gives you that guarantee; what the disclosure has to contain is covered in EU AI Act Article 50. It also does not fit when audio has to stay in Europe. OpenAI processes this model outside the European Economic Area by default, so an assistant that needs EU data residency belongs on a different model. Why that requirement exists is in why voice AI needs European infrastructure, and how we handle it is on the sovereignty page.