The spreadsheet that says self-host is usually right. At real volume, per-minute pricing on a managed platform does become the largest line in a voice budget, and the open-source pieces to replace it genuinely exist and are genuinely good. The spreadsheet is right about the cost it modelled. What it left out is that a voice stack is not a deployment, it is a rota, and the day you own it, you own the pager.
What you are actually replacing
It helps to be precise about the scope, because "self-host the voice agent" usually means self-hosting six things that happen to be adjacent. Each of them has mature open-source options, and each of them is a component you now maintain.
| Layer | What it does | Who is on the hook after you self-host |
|---|---|---|
| Telephony edge | SIP registration, trunks, NAT traversal, media relay. Asterisk, FreeSWITCH or Kamailio territory. | You, and you now also need a carrier relationship. |
| Agent runtime | Session lifecycle, turn-taking, barge-in, worker orchestration. LiveKit Agents or Pipecat. | You, including its upgrade cadence. |
| Speech recognition | Streaming transcription. Whisper variants, or a hosted API you keep. | You, plus GPU capacity if you run it locally. |
| Model access | The language model, self-hosted or via an API. | Shared, but you own the failure behaviour. |
| Speech synthesis | Voices. Piper locally, or a hosted voice provider. | You, and voice quality becomes your product problem. |
| Everything around it | Recording, transcripts, analysis, webhooks, billing data, dashboards, retention. | You, and this is the half people forget to scope. |
That last row is where most estimates go wrong. The interesting engineering is in the first five; the sustained work is in the sixth. A voice platform is mostly the surface around the call: the record of what happened, the hooks into your own systems, the retention policy, the thing you show a customer when they ask what the agent said.
The bill that is not in the spreadsheet
The per-minute saving is real and easy to compute. These costs are equally real and get left out, mostly because they are people rather than line items.
- 01
Someone answers the phone at night
A phone line is a synchronous, business-hours-critical service with no retry semantics. A queue can absorb an outage for twenty minutes; a caller cannot. That means a rota, and a rota needs at least two people who both know the stack, which is a materially different hiring problem from one person who set it up. - 02
The upgrade treadmill does not stop
The agent frameworks in this space move quickly, and voice plugins break in ways unit tests do not catch: a model swap that silently changes turn-taking, a transcription plugin that needs a parameter it did not need last version. Every upgrade needs real test calls, not a green CI badge. - 03
You now own a carrier relationship
Numbers, porting, trunk capacity, fraud controls, and someone to escalate to when the problem is upstream of you. This is a procurement and operations responsibility, not a technical one, and it is nobody's favourite work. - 04
Observability has to be built, not bought
You need per-call traces that span SIP, media and model turns, because that is the only way to answer "why was this one call bad". Generic APM will not give you that. Neither will the conversation-level tooling in this market, which largely stops above the transport layer. - 05
Recovery has to be tested to count
A backup you have never restored is a hope. A failover you have never triggered on purpose is a diagram. Both take deliberate, scheduled, slightly frightening work that always loses to feature work unless someone protects it.
Compare against a fully loaded operating cost
What a 3am voice outage looks like
Abstract on-call cost is easy to wave away, so here is the concrete version. This shape of incident is ordinary rather than exotic, which is exactly the point.
03:12 alert inbound calls failing · 14 in 4 min
03:12 you wake up
03:19 triage SIP registers fine · 200 OK on INVITE
03:24 triage no RTP outbound · media stuck
03:31 found STT worker OOM-killed, restarted, port
rebound behind stale NAT mapping
03:38 fix restart media path, drain workers
03:44 verify test call · two-way audio ✓
03:51 note root cause unclear · will look tomorrow
─────────────────────────────────────────────────
39 minutes · ~90 calls lost · nobody else could
have done it, because nobody else knows the stackThree things about that log are worth sitting with. It took thirty-nine minutes, which is a good outcome, not a bad one. The root cause was still open at the end, meaning the same night can happen again next week. And the reason it took thirty-nine minutes rather than four hours is that the person paged had the whole stack in their head, which is also the reason this system now has a single point of human failure who would like to take a holiday.
That is the actual trade in self-hosting. Not money for control, but money for concentration of knowledge. It works beautifully while your best engineer is engaged and available, and it degrades exactly when they are not.
Capacity is not a server, it is three ceilings
The most common self-hosted surprise is discovering that scaling voice means scaling three unrelated things, and the one you hit first is rarely the one you planned for.
| Ceiling | What it limits | How it fails |
|---|---|---|
| Carrier call rate | New calls per second your provider will accept, separate from how many can be active. | An outbound campaign bursts and the carrier starts refusing setups. Looks like your bug. |
| Trunk channels | How many calls can be simultaneously active on the trunk. | Caller number N+1 gets a busy signal. No error anywhere in your application. |
| Worker compute | Concurrent sessions per worker process, driven by which plugins are loaded. | Sessions degrade before they fail: latency creeps, turn-taking gets sloppy. |
The third one deserves the warning. Published guidance for the common agent frameworks puts a single worker somewhere between a handful and a couple of dozen concurrent sessions depending on what runs in-process, and local voice-activity detection is frequently the component that sets the ceiling. That is a wide range, which means your real number is an empirical question you have to answer with load, on your hardware, with your plugin set. Nobody can hand it to you.
The uncomfortable part is that the compute ceiling degrades rather than trips. You do not get an alert saying capacity exceeded. You get calls that feel slightly worse, an agent that starts talking over people, and a support queue that fills up before your monitoring notices anything is wrong.
Model churn is what breaks it after a year
A self-hosted stack that works in month one usually fails in month twelve for a reason that has nothing to do with infrastructure. The models move. A new realtime model ships with better latency and different semantics. A transcription provider deprecates the version you pinned. A voice you built your brand around gets a successor and the old one goes into maintenance.
On a managed platform that churn is somebody's job. Self-hosted, it is a standing tax on your roadmap: every quarter, someone has to evaluate what changed, run comparison calls, and decide whether to move. Skip it for a year and you are running a stack that sounds a generation old, which callers notice even when they cannot name why.
The hard part of self-hosting is not standing it up. It is still being current in eighteen months, with the same two people, while shipping your actual product.
When self-hosting is the right answer
We build and operate a platform, so treat our view accordingly, but this is a genuine answer rather than a rhetorical one. Self-hosting is clearly correct when at least two of these three hold.
The first is volume that makes the arithmetic overwhelming rather than marginal. If per-minute fees are a rounding error against two engineers, they are not the problem you should be solving. If they exceed a small team's fully loaded cost by a wide margin, the case makes itself.
The second is a hard requirement no hosted arrangement can satisfy: an air-gapped network, a regulator that requires the workload inside your own perimeter, or a data classification that forbids third-party processing regardless of contract. This is a constraint, not a preference, and it settles the question.
The third is that voice is your product rather than a feature of it. If your differentiator is the conversation itself, owning the runtime is strategic and the operational cost is simply the cost of your core business. If voice is one capability inside a larger product, you are about to hire for a speciality that is not your speciality.
Three honest ways out
Most teams reaching for self-hosting want one of three things: lower marginal cost, control, or the absence of lock-in. Those are separable, and only one of them actually requires you to hold the pager.
- 01
Self-host it and staff it properly
The right choice when the conditions above hold. Do it with two people who both know the stack, a real on-call rota, scheduled failover drills, and per-call tracing built before you need it. Budget the operations, not just the servers. - 02
Stay managed and attack the marginal cost directly
If cost is the real driver, most of it is not the platform fee. Bring your own provider keys so you pay them at their rates, shorten the system prompt, shorten the answers, and check whether your billing rounds per second. That is usually a larger saving than the fee you were trying to escape, for a fraction of the work. - 03
Have someone else hold the pager on infrastructure you can take over
This is what we do, so read it as our position rather than as neutral advice: we build and run the orchestration layer, we co-build your voice system with your team, and you get a dashboard and API you own outright. On-premise deployment is available when the constraint is genuinely regulatory, under a licence, scoped as a project rather than switched on from a dashboard. The point is that lock-in and operational burden are separate problems, and you can refuse one without accepting the other.
What we would push back on is treating self-hosting as the default expression of technical seriousness. Running your own voice stack is a real capability and sometimes exactly right. It is also a rota, a carrier relationship, an upgrade treadmill and a quarterly model review, indefinitely, for a component most companies would rather have simply work. Decide it on the fully loaded number, not on the per-minute one.
Sources Component landscape from the public documentation of the common open-source voice stacks (LiveKit Agents, Pipecat, Whisper, Piper, and the Asterisk/FreeSWITCH/Kamailio telephony family); the concurrency-per-worker range reflects those projects' own deployment guidance and varies with plugin set. The incident shape and the operational costs come from running voice infrastructure in production ourselves.