A Voice in the Room

Local voice agents in three shapes — Voicebox, VoxCPM and Dograh placed among Moshi, Unmute, Pipecat, LiveKit and Home Assistant — which pieces are models, frameworks or apps, what each needs, and the household cascade timed on four CPU cores

Author

Behzad Samadi

Published

September 13, 2026

An Agent on Your Own Machine made one promise: what you tell the agent stays on the disk you can unplug. This article is the same promise, said out loud. Speak to the machine in the room, have it answer in a voice, and have nothing — not the audio, not the transcript, not the reply — leave the house. A year ago that meant a hobbyist’s weekend and a robotic voice. In September 2026 it is a crowded field with three distinct architectures, a dozen serious projects, and one stack that ships on a $59 box.

The question that started it named three projects — Voicebox, VoxCPM and Dograh — and asked what else there is. The first finding is that those three are three different kinds of thing: an app, a model and a platform. Most of the confusion in this field is exactly that confusion, so the article sorts the field by shape before it sorts it by name. It was written the way the robotics articles were: everything that could be read at the source was — thirty-odd repositories, their licence files, the package index — and one thing was run: the pieces of a household voice pipeline that fit on a machine with no GPU, timed here. Every latency is labelled as the project’s claim or as this machine’s number.

NoteTL;DR
  1. Three shapes. A cascade (speech-to-text → language model → text-to-speech) runs on a CPU and lets you keep the model you already have. A speech-native model hears and speaks from one network but still takes turns, and wants a GPU. A full-duplex model listens while it talks — Moshi, PersonaPlex, MiniCPM-o 4.5 — and wants a big one.
  2. The three names are an app, a model and a platform. Voicebox is a MIT desktop voice studio — seven TTS engines, Whisper, a small Qwen3 — with an MCP server so a coding agent can speak. VoxCPM2 is a 2B Apache-2.0 tokenizer-free TTS model, the mouth only. Dograh is a BSD-licensed self-hosted alternative to Vapi and Retell whose Docker stack contains no speech model at all; you bring the providers.
  3. The household cascade is fast enough on four cores. Measured here: Kokoro speaks at a third of real time, Whisper tiny transcribes a sentence in half a second and Moonshine in a third, a streaming Zipformer keeps up using 18 ms of every 100, and openWakeWord costs 3.5 ms a frame. The brain — your LLM — is the only stage that will feel slow.
  4. The best voice quality is on the GPU shelf. Kyutai’s Unmute (16 GB VRAM, ~450–750 ms), Moshi (24 GB, ~200 ms on an L4), Qwen3-Omni (~80 GB in bf16). MiniCPM-o 4.5 is the one omni model whose stated floor is a laptop.
  5. Read the weights’ licence, not the code’s. openWakeWord’s code is Apache and its wake words are non-commercial; Piper is GPL; Fish Audio S2 Pro is research-only; PersonaPlex’s weights carry NVIDIA’s terms on top of Moshi’s. Everything else named here is MIT, BSD or Apache end to end.

Three names, three kinds of thing

Voicebox calls itself “the open-source AI voice studio”: “Clone any voice. Generate speech. Dictate into any app. Talk to agents in voices you own.” It is a desktop app — Tauri, with a Rust backend — for macOS, Windows, Linux and Docker, under MIT. Inside it are seven text-to-speech engines (Qwen3-TTS and Qwen’s CustomVoice, LuxTTS, Chatterbox Multilingual and Turbo, Hume’s TADA, Kokoro), OpenAI’s Whisper for transcription “on MLX (Apple Silicon) or PyTorch (CUDA / ROCm / DirectML / CPU)”, and a bundled Qwen3 at 0.6B, 1.7B or 4B that cleans up dictation and applies a persona. The part that makes it an agent story is the MCP server: any MCP client — Claude Code, Cursor, Windsurf — can ask Voicebox to say something in a voice you chose. So Voicebox is the mouth and the ears and the dictation loop; the conversation partner is whatever agent calls it. It is not, on its own, something you talk to.

VoxCPM is a model. “VoxCPM2 is a tokenizer-free Text-to-Speech system that directly generates continuous speech representations via an end-to-end diffusion autoregressive architecture,” from the MiniCPM team at OpenBMB. Three generations in a year — 0.5B in September 2025, 1.5 in December, VoxCPM2 at 2B in April 2026 — 30 languages and nine Chinese dialects, 48 kHz output, voice design from a text description and cloning from a clip. “Weights and code released under the Apache-2.0 license, free for commercial use.” It quotes a real-time factor “as low as ~0.3 on RTX 4090” and streams; pip install voxcpm gives you 2.0.3. It is the best open mouth on the list by the project’s own account, and at 2B it is a GPU mouth.

Dograh is a platform: “the open-source, self-hostable alternative to Vapi & Retell.” A visual workflow builder — start nodes, agent nodes, tools, transitions, end-call outcomes — a test panel, an MCP server so a coding agent can build the workflow for you, and telephony through Twilio, Vonage, Telnyx, Plivo, Asterisk and others. One Docker command; BSD 2-Clause. Its Compose file, read in full, runs Postgres, Redis, MinIO, nginx, coturn, an API, a UI and a Cloudflare tunnel — and no speech model and no LLM. “On Prem, BYOK across Speech to Speech or LLM/STT/TTS” is the honest line: Dograh orchestrates; the voice and the brain are providers you point it at, hosted by default, local if you run them.

Three good projects, and none of them alone is a local voice agent. Put them side by side and the field’s shape appears.

Three shapes

Every local voice agent is one of three things, and the choice decides the hardware, the latency and which licences you have to read.

1. The cascade: STT → LLM → TTS. Three models, three latencies added together, but each one replaceable and the middle one any text model you already run. This is Unmute, Pipecat, LiveKit Agents, Dograh, Bolna, Speaches, Home Assistant’s Assist pipeline, Voicebox’s dictation loop, and every weekend project. Its latency is a design problem — stream every stage and detect the end of the user’s turn early — not a hardware one.

2. Speech-native, half-duplex. One model hears audio and produces audio, usually through a “Thinker–Talker” split, but still waits for you to finish. Qwen3-Omni and Qwen3.5-Omni, MiniCPM-o in its half-duplex mode, Step-Audio 2, Kimi-Audio. Ultravox is half of this (speech in, text out); Sesame’s CSM is the other half (text and audio context in, speech out). Better prosody and paralinguistics than a cascade, because nothing is flattened to text on the way through; a GPU, because the model is the LLM.

3. Full-duplex. The model listens while it speaks — interruptions, back-channels, “mm-hm” — from a single network with two audio streams. Moshi (Kyutai, 2024), NVIDIA’s PersonaPlex (January 2026, built on Moshi), MiniCPM-o 4.5 in full-duplex mode. The rarest, the most natural, and the most GPU-bound.

The cascade is the only shape that runs on a CPU today, so it is the one this article measures. The other two are described with the floors their authors state.

Shape one: the cascade

Unmute is Kyutai’s reference cascade: “a system that allows text LLMs to listen and speak by wrapping them in Kyutai’s Text-to-speech and Speech-to-text models.” Kyutai STT, any LLM served by vLLM, Kyutai TTS, a Rust backend, a web front end, Docker Compose, MIT. The requirement is “a GPU with CUDA support and at least 16 GB VRAM. Architecture must be x86_64.” Latency “decreases from ~750ms when running everything on a single L40S GPU to around ~450ms” with a GPU per service; Kyutai’s own page says the whole response “is below a second” (search). The underlying models — streaming STT at 1B (“0.5 second delay”) and 2.6B, TTS at 1.6B and about 220 ms — are built on what Kyutai calls Delayed Streams Modeling, ship for PyTorch, MLX (“iPhone/Mac”) and Rust, and are CC-BY 4.0. Unmute is what a cascade looks like when one lab owns every stage.

Pipecat (1.10.0, released 12 September 2026, BSD-2) is the framework you reach for when you want to own the stages yourself: “an open-source Python framework for building real-time voice and multimodal conversational agents,” from Daily. Its service list names the local options explicitly — Whisper for STT, Ollama for the LLM, Kokoro and Piper for TTS, Silero for voice activity — next to a dozen cloud providers each, and the PyPI extras add pocket-tts, mlx-whisper, moonshine and local-smart-turn. There are 52 extras; pip install pipecat-ai alone gives you a framework that can neither hear nor speak, which is the first trap. Pipecat’s own end-of-turn model, Smart Turn v3.2, is the interesting piece: “Whisper Tiny as a base, with a linear classifier layer,” about 8M parameters, “runs in as little as 10ms on some CPUs,” 23 languages, BSD-2 for data, training script and weights. It answers the question a VAD cannot — not “has the audio gone quiet?” but “has the person finished?”

LiveKit Agents (1.8.1, Apache-2.0) is the other framework, built around LiveKit’s WebRTC server: “realtime, programmable participants that run on servers,” with “a comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API,” and “fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server.” Its plugins are mostly cloud names; local speech goes through an OpenAI-compatible server, and its turn detector — “a custom open-weight model” that “requires <500MB of RAM” and “runs on CPUs with modest system requirements” — is under “the LiveKit Model License,” not Apache. Read it before you ship it.

Speaches is the adapter that makes the previous two local: an “OpenAI API-compatible server supporting streaming transcription, translation, and speech generation,” faster-whisper behind the transcription endpoint, Kokoro and Piper behind the speech one, a Realtime API, CPU or GPU, Docker, MIT. Anything that can talk to OpenAI’s audio endpoints can talk to Speaches instead, which is most things.

Bolna (MIT) is a cascade built for phones — Deepgram or Azure for STT, Polly, ElevenLabs or Cartesia for TTS, Twilio or Plivo for the call — with the LLM as the one slot that can be “hosted via VLLM.” Cloud-first, with a local brain. Dograh sits beside it, as above: the same shape with a visual builder and a longer telephony list.

The one that runs on a $59 box

The largest deployed local voice stack is not any of the above. It is Home Assistant’s Assist pipeline, and it is worth its own section because it is the only one whose design starts from the assumption that the server is a small computer in a cupboard.

The glue is Wyoming: “a peer-to-peer TCP protocol for voice assistants (basically JSONL + PCM audio),” “an open standard from the Open Home Foundation.” Wake-word services (openWakeWord, microWakeWord, Porcupine), speech-to-text (faster-whisper, whisper.cpp, Vosk), text-to-speech (Piper) and the satellites in each room all speak it, so any piece can be swapped without touching the others. Two of the pieces are clever in the way small systems have to be. Speech-to-Phrase “instead of answering the question ‘what did the user say?’, … answers ‘which of the phrases I know did the user say?’” — it is personalised with your device and area names, and it is what makes a Raspberry Pi 4 usable for “turn off the kitchen lights.” microWakeWord trains wake-word models small enough for TensorFlow Lite Micro, so the detection runs on the microcontroller in the room and the network only wakes for real requests.

That microcontroller is the Home Assistant Voice Preview Edition (search): $59, an ESP32-S3, an XMOS chip for echo cancellation and noise removal, two microphones, a speaker, a hardware mute switch, a dial. Wake word on the device; everything else over Wi-Fi to the server, for which “an Intel N100 or higher is recommended” if you want speech fully local. The 2026 community stack is openWakeWord or microWakeWord, Whisper, Home Assistant’s intent matcher or Ollama with a small Qwen3, and Piper or Kokoro. It is a cascade, and it is the one that has been in people’s kitchens for two years.

Shape two: one model that takes turns

Qwen3-Omni is the flagship: “natively end-to-end multilingual omni-modal foundation models” that “process text, images, audio, and video, while delivering real-time streaming responses in both text and natural speech.” A 30B mixture-of-experts with 3B active, “MoE-based Thinker–Talker design,” 211 ms end-to-end audio latency by its own measure, 119 text languages, 19 spoken in and 10 spoken out, Apache-2.0 end to end. The floor is “78.85GB (15s video) to 144.81GB (120s video)” of GPU memory in bf16 for the Instruct model. Qwen3.5-Omni (30 March 2026) (search) keeps the shape and adds search and function calling from inside the omni model. This is the model to use if you have the hardware and the model to read about if you do not.

MiniCPM-o 4.5 is the one that changes the hardware conversation. A 9B model with “end-to-end omnimodal interaction with streaming video and audio inputs plus text and speech outputs,” open-sourced 3 February 2026, code Apache-2.0. Half-duplex speech needs an “Apple M3/M4/M5 chip with at least 16GB RAM”; full duplex, where “output streams (speech and text), and the real-time input streams (video and audio) do not block each other,” needs an “Apple M4 Max chip with at least 24GB RAM or low-resource Nvidia GPU with at least 12GB GPU memory.” It is the only omni model in this article whose stated floor is a laptop, and it is from the same team as VoxCPM. The model licence was not readable from here; check it.

Step-Audio 2 mini, Kimi-Audio (7B, on Qwen2.5) and Ultravox (on Llama 3.3, “takes in audio and emits streaming text”) fill out the shelf: all speech-in, GPU-hosted, Apache or MIT for code, and each with a different answer to what comes out — audio, audio, text. Sesame CSM (1B, Apache-2.0) is the mirror image: “a speech generation model … that generates RVQ audio codes from text and audio inputs,” a Llama backbone and a Mimi decoder, explicitly “not a general-purpose multimodal LLM” — you pair it with one. The Maya and Miles voices in Sesame’s demo run on an 8B that was not released (search).

Shape three: full duplex

Moshi is where the shape was invented in the open. “A speech-text foundation model and full-duplex spoken dialogue framework,” built on Mimi, a codec that takes “24 kHz audio down to 12.5 Hz” at 1.1 kbps with 80 ms of latency, and a 7B temporal transformer that models the user’s stream and its own stream at once. Theoretical latency “160ms (80ms for the frame size of Mimi + 80ms of acoustic delay),” “~200ms on L4 GPU” in practice. Three implementations: PyTorch (24 GB of VRAM unquantised), MLX for Apple silicon at 4-bit, 8-bit or bf16, and Rust for production. pip install -U moshi_mlx and python -m moshi_mlx.local -q 4 is the one full-duplex command in this article that runs on a Mac. Code MIT and Apache; weights CC-BY 4.0.

PersonaPlex is NVIDIA’s January 2026 build on the same architecture: “a real-time, full-duplex speech-to-speech conversational model that enables persona control through text-based role prompts and audio-based voice conditioning.” 7B, “based on the Moshi architecture and weights,” a --cpu-offload flag for smaller GPUs, code MIT and weights under NVIDIA’s Open Model License. The 240 ms interruption latency and the community MLX ports are from reports, not from here (search).

MiniCPM-o 4.5, above, is the third, and the one with a laptop floor.

Full duplex is the shape that feels like a person — you can cut it off, it can say “right” while you are still talking — and it is the shape where the licence, the memory and the language list are all tighter than the cascade’s. It is also the shape where the LLM is not yours to choose: Moshi’s brain is Helium, PersonaPlex’s is the same, and neither is the model you tuned last month.

The parts, if you assemble your own

A cascade is only as good as its pieces, and in 2026 the pieces are a market of their own.

Ears. Whisper is still the default, through three runtimes — whisper.cpp (pywhispercpp), CTranslate2 (faster-whisper), MLX (mlx-whisper) — with large-v3-turbo the usual size. The alternatives are the story of the year. Moonshine (MIT) is built for exactly this use: “an open source AI toolkit for developers building real-time voice agents,” “everything runs on-device,” models from 1 MB up, streaming. It ran here. Kyutai STT streams with a semantic end-of-speech signal. Voxtral Realtime (search) — Mistral, 4 February 2026, Apache-2.0, 4B — has a configurable delay from 80 ms to 2.4 s and Whisper-level accuracy at 480 ms. Qwen3-ASR (0.6B and 1.7B, 52 languages, Apache-2.0) and NVIDIA’s Parakeet (CC-BY) are the accuracy leaders at small sizes (search). And Gemma 3n (search) makes “a token for every 160ms of audio,” so a 2–4 GB general model can be its own ears.

Mouth, smallest first. Kitten TTS at 15M, 40M and 80M parameters on ONNX, “runs efficiently without a GPU,” Apache-2.0. Kokoro, 82M, Apache-licensed weights, nine language variants — ran here at a third of real time. Pocket TTS from Kyutai, 100M, “~6x real-time on a CPU of MacBook Air M4,” “~200ms to get the first audio chunk,” cloning from a WAV, MIT, training code released in August 2026. Piper, the Home Assistant default, now GPL-3.0-or-later on PyPI. Chatterbox from Resemble — a 0.5B base, a Multilingual V3, a 350M Turbo “for low-latency voice agents” and a 110M Nano that “runs on CPU at 3x faster than realtime on 8 cores” — MIT, with a perceptual watermark in every output. Qwen3-TTS (0.6B and 1.7B, “first-packet latency as low as 97ms,” ten languages, Apache-2.0, January 2026). VoxCPM2 (2B, above). Then the GPU-only and the special-purpose: Microsoft’s VibeVoice for 90-minute four-speaker audio and a 0.5B Realtime at ~300 ms; Orpheus on a Llama-3B backbone with emotive tags and a llama.cpp path; Dia for two-speaker dialogue with no CPU support yet; Hume’s TADA, a 1B and 3B “unified speech-language model” whose weights carry Llama 3.2’s licence; Fish Audio S2 Pro, 4B and 80 languages, under a research licence that is not open for commercial use.

Turn-taking. Silero VAD — about 2 MB, “one audio chunk (30+ ms) takes less than 1ms to be processed on a single CPU thread,” MIT — is in nearly every stack, and ran here. Above it sit the semantic detectors: Pipecat’s Smart Turn (BSD-2) and LiveKit’s turn detector (its own licence), both CPU-sized. The difference between a good voice agent and an irritating one is mostly which of these you use and how you tune the silence.

The doorbell. openWakeWord: pre-trained “alexa, hey mycroft, hey jarvis, hey rhasspy,” under 0.5 false accepts an hour, “a single core of a Raspberry Pi 3 can run 15-20 openWakeWord models simultaneously in real-time,” trained “with 100% synthetic speech.” It ran here and fired on a synthetic voice. Code Apache-2.0; the pre-trained models CC-BY-NC-SA 4.0 — the one non-commercial licence in the household stack, and the one people miss. microWakeWord for the microcontroller.

The runtime. sherpa-onnx (1.13.8, Apache-2.0) runs streaming and offline STT, TTS, VAD, keyword spotting, speaker identification and enhancement, “locally,” on Linux, macOS, Windows, Android, iOS, Raspberry Pi, RISC-V boards and in WebAssembly, with bindings in twelve languages. It is the engine under a good fraction of the projects above, and it is what this article’s run used, because its authors mirror their models as GitHub release assets and Hugging Face was out of reach.

The closed baselines

For scale (search): OpenAI’s gpt-realtime is one model, audio in and out, at $32 per million audio input tokens and $64 per million output, around 800 ms voice to voice when configured well. Google’s Gemini Live API is native audio with thirty voices in twenty-four languages. Both are what Dograh, Pipecat and LiveKit list as “speech-to-speech” providers. Neither runs in the room, and the whole of this article is the price of that sentence.

What ran here, with no GPU and no hub

The machine is the four-core, GPU-less container from the last two articles, and the model hub was blocked again, so the experiment was the household cascade minus its brain: can the ears, the mouth, the turn-taking and the doorbell run on a CPU, and how fast?

There is no microphone in a container, so the script makes its own speech. Kokoro reads a sentence — “The quick brown fox jumps over the lazy dog, and the meeting moves to half past nine on Thursday” — and that 5.7-second clip goes to everything that listens. A second clip, “Hey Jarvis, what is the weather like today?”, goes to the wake-word detector. The script is seventy lines; the run note has every number.

Stage Work Result
Kokoro, 82M, ONNX 1.79 s for 5.70 s of audio — RTF 0.31 a natural sentence
Silero VAD, ONNX 57 ms for 5.7 s two segments, split at the comma
Whisper tiny.en, int8 0.50 s per sentence every word right; “Quick Brown Fox” capitalised
Moonshine tiny, int8 0.29 s per sentence every word right
Streaming Zipformer, 100 ms chunks 18 ms of compute per 100 ms of audio; 0.18 s to flush every word, upper case, no punctuation
openWakeWord “hey jarvis” 3.5 ms per 80 ms frame score 0.998, fired as “Jarvis” ended

The whole script — five model loads included — took twenty seconds. What the table says: the mouth is three times faster than real time and streams by sentence, the ears are ten to twenty times faster than real time for whole utterances, the streaming recogniser idles four fifths of the time, and the wake word is free. Add the parts of the cascade this machine could run and they come to well under a second. The brain is the budget: put a 4B model behind this on the same four cores and it will be the only stage you wait for, which is the argument for a Mac with unified memory, or for MiniCPM-o’s 16 GB laptop floor if you want one model doing everything.

Three traps, briefly, because they will happen to you too. A truncated model download reads as Protobuf parsing failed rather than as a short file. openWakeWord’s current release pins a TensorFlow Lite package with no Python 3.12 wheel, so the resolver silently hands you the 2023 version with a different API. And a streaming transducer needs a second of silence to finish its last word — which is what the VAD is for.

What did not run: anything hosted only on Hugging Face — VoxCPM2, Pocket TTS, Moshi, Qwen3-TTS, Chatterbox, Smart Turn — so their speeds above are their authors’; any LLM, so there is no end-to-end figure; any GPU shape; and any real microphone in any real room.

What a local voice agent wants from you

In the order you will meet them.

  • A decision about shape. CPU or small Mac: the cascade, with the model you already run in the middle. A 16 GB Mac: MiniCPM-o 4.5 half-duplex, or the cascade with a better mouth. A 24 GB Mac or a 12 GB GPU: Moshi at 4-bit, or MiniCPM-o full-duplex. A 16 GB CUDA card: Unmute. Anything larger: Qwen3-Omni.
  • A turn detector, not just a VAD. Silero tells you the room went quiet; Smart Turn tells you the person stopped. The second is the difference between an assistant and an interruption.
  • Streaming at every stage. A cascade that waits for the whole transcript, then the whole reply, then the whole audio, adds three latencies. One that streams adds three first-token latencies. Pipecat and LiveKit exist to make the second the default.
  • The licence of the weights. Code licences are nearly all permissive. The weights are where the exceptions live — openWakeWord’s models, LiveKit’s turn detector, PersonaPlex, TADA, Fish — and a product is built on weights.
  • A microphone that is not a laptop’s. Echo cancellation is why the $59 box has an XMOS chip. If the agent hears itself, no model saves you.

What to do this week

  • Run one mouth and one ear, no framework. pip install kokoro-onnx sherpa-onnx, the two Kokoro files and one sherpa-onnx recogniser from their GitHub releases, and the seventy-line script from the notes. You will have the numbers for your own machine in ten minutes.
  • Try Moshi on a Mac if you have 16 GB: pip install -U moshi_mlx, python -m moshi_mlx.local -q 4. It is the fastest way to feel what full duplex is, and to decide whether you need it.
  • Pick the framework by its local extras. pipecat-ai[kokoro, moonshine, local-smart-turn, silero] and Ollama is a complete local cascade in one Python file; LiveKit plus Speaches is the same with a WebRTC front door.
  • Point Voicebox at your coding agent. If what you want is an agent that talks rather than one you talk to, its MCP server is a one-line addition and the seven engines are already installed.
  • If you have a Home Assistant server, skip all of the above and add a Voice PE. It is the one local voice agent that a household will use.

Sources

Research notes — the pages read, the PyPI records, the run and the pages that could not be fetched — are in the accompanying folder.