Speech to text inference

Speech-to-text (STT / ASR) inference turns audio into text locally, as an API service instead of a cloud endpoint. One constraint shapes the whole setup on doombox: the RTX 3090 is dedicated to LLM inference (llama-swap (OSS) holds ~23 GB of 24 GB VRAM), so STT must run on CPU and still be fast enough for interactive voice-message transcription.

What runs now

voiceapi — Python/FastAPI service on :5092 (fronted by nginx as stt.h.krzysztofkaminski.dev), exposing the OpenAI-compatible /v1/audio/transcriptions endpoint.

• Model: NVIDIA Parakeet TDT 0.6B v3, INT8-quantized ONNX (~640 MB on disk) — multilingual, 25 European languages including Polish, with auto-detection.
• Inference: ONNX Runtime, CPU-only, 8 intra-op threads / 1 inter-op; Voice Activity Detector chunks audio by detected speech instead of fixed-size windows.
• Upstream: groxaxo/parakeet-tdt-0.6b-v3-fastapi-openai (itself a fork of Shadowfita’s v2 service), pinned at upstream HEAD — the only local changes are a uv pyproject.toml and a CPU .env.
• Footprint: ~880 MB RSS, systemd-capped at 4 GB RAM and 80% CPU.

How it is consumed

Hermes Agent points its stt.openai config at http://localhost:5092/v1, so transcribing a Telegram voice message is a plain OpenAI client call: OGG/Opus upload → server-side decode → VAD chunking → ONNX decode → text. A local Hermes patch raises the STT HTTP timeout to 300 s, because CPU transcription of a 5-minute brain-dump takes ~170 s and the stock 30 s timeout truncated it.

Why Parakeet on CPU

Parakeet TDT 0.6B v3 is among the fastest ASR models for CPU inference — roughly 20–30x real-time with INT8 — and a drop-in replacement for OpenAI’s Whisper API surface: no API key, no per-minute cost. The TDT (token-and-duration transducer) architecture predicts how many audio frames each token covers, skipping padding frames — that is why it outruns similarly-sized CTC/RNNT models on CPU. GPU placement was rejected for the same reason as every other audio service on this host: the GPU belongs to llama-swap, and 20–30x real-time on CPU already exceeds what voice-message transcription needs (decision 2026-06-17).

The parakeet.cpp alternative — and the port work

parakeet.cpp (OSS) is a C++/ggml port by the LocalAI team: single static binary, GGUF quants (q8_0 ~250 MB vs ~640 MB ONNX), optional CUDA, transcripts validated at WER 0 against NeMo. It was rejected in the 2026-06-17 decision when it was three weeks old and WAV-only — Telegram voice messages are OGG/Opus, and the built-in server was labeled “example, not production”.

That gap is what the local port branch closes: 13 commits on feat/ffmpeg-fallback add audio format sniffing (magic bytes), vendored MP3 (dr_mp3) and OGG/Vorbis (stb_vorbis) decoders, an ffmpeg runtime fallback for Opus and unknown formats, and server-side acceptance of mp3/ogg uploads — the exact blocker cited in the rejection. The benchmark that would justify the switch (latency, real-time factor, peak RAM across 5 s → 10 min inputs vs voiceapi) is still unmeasured; homelab roadmap §5 tracks it.

transcribe.cpp (OSS) is the wider horizon: 16 STT model families (Canary, Granite Speech, Moonshine, Nemotron ASR streaming) on ggml with streaming and diarization. If the audio stack consolidates on ggml, it subsumes parakeet.cpp’s role.

Whistle (2026-10-02) — the tiny multilingual challenger, Polish included

needle (OSS) (cactus-compute, Python, MIT) ships Whistle: STT in a 16.9 MB file, CPU-only, in the same engine as their Needle LLM decoder (.cact container, 2-4-bit Cactus Quants, SIMD kernels, shared KV cache). Numbers from the announcement thread (clippings/x-2106083041265562075-thread.md): 4.31 WER LibriSpeech test-clean / 10.49 test-other vs Whisper base’s 4.9/11.0 at 145.3 MB — 9× smaller, 6× faster; FLEURS avg 21.4 vs 24.5. Languages: EN/DE/FR/ES/IT/NL/PL — first small-model option in this comparison with Polish, which matters for the Polish-speaking household lane. Keyword biasing rescues names (“Krzysztof”) during beam search; word timestamps from decoder attention; silence returns empty instead of hallucinated text (the classic Whisper failure mode). 30 s per pass, 17 platforms incl. WASM; pip install cactus-needle, needle whistle compare A/Bs against Whisper/Moonshine with timings — i.e. it ships its own benchmark harness for exactly the parakeet-vs-voiceapi comparison this note still owes. Caveats vs the incumbent: 30-second cap per pass, unknown streaming story, no diarization.

Blog capture: clippings/cactuscompute-com-blog-whistle.md.

Open questions

• Does parakeet.cpp replace voiceapi once benchmarked, or does the mature Python stack (any input format, web UI, progress polling) keep winning?
• Does the Nemotron streaming ASR generation change the streaming/realtime calculus, or is batch transcription all this homelab needs?
• Will diarization land in the local stack via transcribe.cpp (OSS)‘s ggml diarizers, or is single-speaker transcription all this homelab needs? See Diarization.

Links

• Voice Activity Detector
• parakeet.cpp (OSS)
• transcribe.cpp (OSS)
• llama-swap (OSS)
• Whisper (red — future note)

The studio-tier aggregator: VoiceStudio

VoiceStudio (OSS) (debpalash, 53.4k★, AGPL-3.0, Python/Electron; +14.7k stars in one week per @trending_repos) — “the open-source, fully-local ElevenLabs alternative”: voice cloning, voice design (describe a voice, it’s synthesized), video dubbing, dictation, transcription, and audiobook creation, in 646 languages. Local workflows run on your hardware (remote services optional, consent-gated analytics); Electron app + Docker; local model manager workspace. Relevance to the local-speech lane: it’s the productization layer over the exact model families this note tracks — the doombox already runs Parakeet STT (:5092) and llama-swap TTS; VoiceStudio is what a full studio stack around those primitives looks like. AGPL-3.0 (notable for an Electron product); watch the local-model manager’s model list as a curated index of the current best open speech models.

The output side: Paradee, Kokoro distilled to 8M (2026-10-07)

This note is STT-centric, but the speech lane has an output side too: HuggingApps — “Kokoro TTS, but 10× smaller”: Paradee (paradee (OSS), hand-registered — a Space, not a repo) distills Kokoro-82M → 8M params, 9 MB of weights, “speaks faster than real time on a single CPU thread and runs on your toaster.” Same family math as Parakeet on the input side: when the model is 9 MB, synthesis becomes a library, not a service — the Small language models thesis for speech, both directions.