Offline Voice Assistant on a Local LLM

Author: Jason Wolff, STL Net Services · September 2026 · Apache License 2.0

Download local-ai-voice-assistant.zip: the installer, the voice bundle, the optional neural-voice stage, LICENSE and NOTICE. The full incident log is in its README.

Say "computer", ask a question, and a small box on your desk answers out loud. It needs no cloud account and no GPU, and nothing leaves the machine except the web searches you ask for. Unplug the internet and it still answers. You just lose web search.

It installs onto a fresh Ubuntu machine in three stages. It was built and debugged on an Intel NUC10i5 (8 threads, 15GB RAM) and an i5-9400 desktop.

What it installs

Piece	What it is
llama-server	ik_llama.cpp built with AVX2, CPU inference on loopback :8080. Default model is Granite 4.0 H Tiny Q4_K_M (MoE, 7B total / 1B active, ~4.2GB)
opencode-server	Persistent agent API on loopback :4096, with a dedicated voice agent
MCP tools	search_web, rag_query, rag_ingest_*, get_current_time, system_info
SearXNG	Docker, loopback-only, backs search_web
voice-assistant	Wake word "computer", Silero VAD, faster-whisper small.en, Piper TTS
qwen-tts	Optional neural voice with per-reply tone control, loopback :8081
lab-home	Landing page, live dialog log and machine control on port 80
glances	System monitor web UI on port 61208
OpenWebRX + RTL-SDR	Browser radio receiver for an RTL-SDR USB dongle, port 8073

Stage 1 also turns on unattended security updates and a daily ClamAV scan. opencode is installed system-wide, so any user on the box can use it with no setup.

The model, agent, search and TTS services listen on loopback only. Three things are reachable from your LAN on purpose: lab-home (port 80), glances (61208) and OpenWebRX (8073). See the warning below.

Requirements

• Ubuntu 22.04 or 24.04 with sudo
• An AVX2 CPU. The installer checks and refuses without it.
• 12GB+ RAM (16GB if you want the neural voice) and ~20GB free disk
• A microphone and speakers for the voice stages

Install

unzip local-ai-voice-assistant.zip && cd local-ai-voice-assistant

# Stage 1: model, inference, agent, search, RAG. Run as YOUR user, not sudo.
./install.sh

# Stage 2: the voice assistant and web console (~10 min, large pip download)
sudo bash voice/voice-setup.sh

# Stage 3, optional: neural voice with tone control (~1.4GB of weights, builds from source)
sudo bash voice/qwen-tts-setup.sh

# Optional: wifi. Without it the box is off the network when the cable is out.
sudo ./wifi-setup.sh 'SSID' 'password'

Log out and back in once after stage 1 so group membership applies. The model download resumes if it's interrupted and is checked against its published SHA-256. If you already have the same GGUF on your network, copy it to /opt/ai/models/ first and the download is skipped. Settings such as MODEL_REPO, MODEL_FILE, CTX_SIZE and AI_ROOT can be overridden with environment variables. The README lists them all.

Each stage ends by checking itself. Stage 2 sets up the mic for the detected codec and fails loudly if a real capture is silent, then runs the logic tests and times a real transcription under the service's CPU quota.

How it talks

Three speech engines, picked automatically:

Engine	Real-time factor	Used for
Qwen3-TTS (neural, via qwentts.cpp)	~2.8	Answers, when installed
Piper	~0.03	Automatic fallback if the neural voice fails
espeak-ng (formant)	~0.003	Status lines, and every reply in basic mode

The neural voice is slower than real time, so the assistant fills the gap. It says "Received. Working." once it has heard you, then "Ready. 4 parts, about 90 seconds." before it speaks. These lines use a deliberately different 1979-style Klatt voice, so you never mistake them for the answer.

Basic speech mode puts everything on espeak-ng. Turn it on from the console, or let it turn on automatically when free memory drops below 1200MB. It announces the switch both ways. That's the difference between answering and swapping on a 16GB box.

At boot it reads out its own console address digit by digit. It tells you separately when there's no internet (search is unavailable) and when there's no network at all (no console either).

The web console

http://<box>/ has three pages: a landing page with a mic level and source panel, /dialog with everything the assistant heard and said, and /control:

• free memory, LAN address, internet reachability and the state of all five services
• basic speech mode on or off, which applies from the next thing you say with no restart
• restart the assistant or the TTS service
• shut down or reboot, which are POST-only and need a confirm field so a link prefetch can't power the box off

Three pages are open to your LAN

• The console (port 80) has no login. Anyone who can reach it can read the dialog log and shut down or reboot the box.
• glances (port 61208) has no login and shows CPU, memory, processes and network.
• OpenWebRX (port 8073) creates its admin account through a first-run wizard. Whoever visits it first sets the password, so open it yourself straight after installing.

Keep the box on a trusted network or firewall these ports. The installer's closing summary explains how to bind glances and OpenWebRX to loopback instead.

Lessons that made it usable

The zip's README documents more than 30 incidents with root causes. These made the biggest difference:

• --threads $(nproc) cost 15x. nproc counts logical CPUs, so llama.cpp ran 8 threads on 4 cores: 0.54 vs 11.75 tokens/sec. Threads now come from lscpu physical cores.
• Whisper ignored the CPU quota. faster-whisper sized its pool from the machine's cores while systemd's CPUQuota allowed two. Transcribing the single word "computer" took 10.8s instead of 2.6s, and it looked exactly like the model being too big. Now the thread count is derived from the process's own cgroup.
• Every spoken turn cost 78 seconds of prefill. The default agent's coding-tool schemas made a 5,136-token prompt. A lean voice agent and one reused session brought turns down to 1.6–5.9s.
• The wake word fired on podcasts. "computer" in text matched "computer vision" mid-sentence. The matcher is now start-anchored with a length limit and has a regression test that was run against 675 real transcripts.
• Markdown reached the speakers. Piper read "asterisk asterisk" aloud. The prompt forbids markdown and the output is stripped anyway, because small models emit it regardless.
• It read ads as news. Search snippets for broad queries are mostly marketing copy, so they're filtered and outlet names are withheld from the model.

Known limitations

• No barge-in. Capture stops while it speaks, so you can't interrupt it.
• One microphone at a time on the onboard codec. A USB mic is the way to concurrent inputs.
• English only.
• Response time. The first turn of a fresh session pays about 2 minutes of CPU prefill. After that, replies start in seconds.

License and attribution

Copyright 2026 Jason Wolff, STL Net Services. Licensed under the Apache License 2.0. You may use, modify and redistribute it, including commercially, as long as the copyright and the NOTICE file's attribution survive in every copy or derivative work. The license grants no rights to the author's name or marks.

Written by Jason Wolff, with Claude (Anthropic) and GLM.