Local LLM inference
Running frontier-class models on your own metal — the doombox (3090/24 GB, 32 GB RAM) is the reference constraint. The cluster’s STT sibling is Speech to text inference; the roadmap home is the homelab Local LM roadmap.
The stack
- Serving: llama-swap (OSS) — one GPU, one model hot at a time, 59 registered entries; the Hermes
irisprofile points at it. QMD/kbd tooling and the gateway ride on top. Engine-level details and the llama.cpp/vLLM/SGLang/lucebox/Strata comparison live in Inference engines. - Quantization: GGUF imatrix quants (Q4_K_M the workhorse, Unsloth/bartowski as the sources) — the Qwen model family is entirely GGUF-served. DFlash2 speculative decoding (+35-70%, lossless) pairs a Q2_K block-diffusion drafter with the Q4 target.
- Engine-level offloading — the frontier of “what fits”: Strata (OSS) (C++, 671★) runs Qwen3.8-Flash-Next (125B total / 6B active) on one 12-24 GB consumer card + 64 GB RAM: GPU holds the dense part + hottest experts, CPU computes the rest of the 512 experts concurrently, a 51B n-gram embedding lookup lives on SSD (read a few rows per token), and MTP speculative decoding checks several guesses at once (1.6-1.8x, exact same output). 60-95 tok/s on an RTX 5070 — faster than reading speed. The lesson: MoE + n-gram embeddings + speculative decoding make parameter count the wrong mental model; active params + offloadable state determine what a gaming PC can run.
Decision rules (from the roadmap + captures)
- Model choice: match activated-parameter budget to the card (ThinkingCap-Qwen3.8-27B Q4_K_M = 17.4 GB fits the 3090 exactly); context costs KV cache — q4_0 KV for 245k.
- Speculative decoding is free speed when a compatible drafter exists (DFlash2); verify losslessness claims, not just tok/s.
- Quantization fidelity reporting matters: trust model cards that publish PPL delta + top-1 agreement + KLD vs BF16 (OrcaRouter/ThinkingCap style), not just “Q4 is fine”.
- Offloading tiers (GPU → RAM → SSD) buy model size at bandwidth cost; Strata’s numbers show the crossover is now usable for MoE with big expert counts.
- KISS check before scaling: if one 27B at Q4 with a drafter does the job, a 125B offload rig is a want, not a need — the roadmap’s escalation order is prompt → RAG → small local → big local.
Related: Small language models (the model shelf) · Qwen model family · DFlash2 · ThinkingCap-Qwen3.8-27B · Strata (OSS) · Uncensored models · Speech to text inference
Translation lane: Index-Translate (the first local-first translation model family)
Index-Translate (OSS) (Bilibili, 662★; via @TeksEdge — “something useful beyond another coding chatbot”) — multilingual translation family built on Qwen3.5, 150 languages, in three sizes: 2B / 9B / 35B-A3B (preview), with official GGUFs for llama.cpp (released 2026-10-03) alongside FP8 and Blackwell NVFP4 for vLLM. The instruction-following is the differentiator — it follows translation instructions: terminology glossaries, tone/style requirements, formatting preservation (JSON, placeholders), community expressions/memes. The family extends beyond text: Index-Echo (translated subtitles/speech in the source speaker’s voice), Index-Homura (syllable-count-controlled translation — songs/lyrics), Index-NativeLong (full-document with cross-passage context). Free OpenAI-compatible API for the 35B-A3B preview (index-translate.bilibili.com/v1) for try-before-download; four benchmarks released with datasets (instTrans, IFMTBench, MEME, …). The KRZ fit: Polish↔English daily reality, docs/quotes translation in vault workflows — a 2B GGUF slot in llama-swap for glossary-constrained translation is a real utility slot (quality must be tested on own documents first, as the tweeter warns).
Hy-MT2: the challenger — 440 MB on-device, 33 languages (2026-10-09)
Hy-MT2 (OSS) (Tencent-Hunyuan, 1,161★; via @TencentAI_News quoting @silvanrec’s viral Spanish post — tweet + repo captured): the “fast-thinking” multilingual translation family in 1.8B / 7B / 30B-A3B (MoE), 33 languages. The headline is the on-device lane: AngelSlim 1.25-bit extreme quantization puts the 1.8B at 440 MB with 1.5× faster inference — GGUF straight to llama.cpp (official 1.25bit/2bit/FP8 quants), “100% on your device, airplane mode, texts never leave your phone.” Claims: 7B/30B beat DeepSeek-V4-Pro and Kimi K2.6 in fast-thinking mode; the 1.8B “surpasses mainstream commercial APIs (Microsoft, Doubao) overall.” Also open-sourced: IFMTBench (translation instruction-following benchmark — the same eval Index-Translate shipped against) and an official Hy-MT2-Translator Skill on ClawHub/SkillHub. The prompt table is the practical gold: canonical instructions for default translation, terminology glossaries, style constraints, personalization preferences, delimiter preservation (“retain the exact same number of delimiters, do not omit, escape, or translate”), and structured-data translation with hard rules (translate only user-facing values, never keys/tags/placeholders). Vs the lane leader: Index-Translate has the breadth (150 languages, speech/song/document variants), Hy-MT2 has the pocket (440 MB beats everything; llama.cpp-native; community already shipping ComfyUI nodes, Pinokio one-click, Docker desktops). KRZ fit: same utility slot, smaller rent — a 1.8B-1.25bit GGUF in llama-swap costs ~0.5 GB VRAM; the shared IFMTBench eval makes the two directly comparable on own documents.
The serve-side surgery lane: Bonsai 2 262k on 12 GB (2026-10)
CaryPalmerr + repo (28★ Python; 2-tweet thread + GGUF on HF, all captured): a 1.58-bit ternary 27B (PrismML PTQ1_0, 5.9 GB) serving the full 262,144-token window with q8_0 KV on a 12 GB RTX 4070 — every improvement made without touching the weights:
- Tiered KV cache (
--kv-vram-cells): first ~113k positions in VRAM, the rest (~5.2 GB) in pinned system RAM mapped into the CUDA address range — bit-identical output; decode 83→14.7 tok/s across the window, and “past 112k” stays usable rather than impossible. - MTP drafting at every depth (
--spec-draft-window; draft cache keeps last 16k rows) — the old 24k cutoff gone. - Harness-proofing, the agentic finding: the model wasn’t broken, the serve contract was — apps sending
effort: "high"got HTTP 500 (HumanEval 0/164 → 160/164 once the server normalizes instead of crashing); token caps cut it off mid-think; tool calls as JSON-in-content failed 8/9 → 9/9 with native XML + server grammar. “After a week of experiments I found the two places agentic work broke down in the model weights — instead of adjusting the weights, I baked the solution into the serve.” AppWorld 64.3% → 70.8% same weights.
Two lessons for this note’s lane: (1) quantized ternary models are serve-bound, not quality-bound — the wins come from KV placement, speculative depth, and contract tolerance in llama.cpp forks, i.e., the same strata/Strata layer this note tracks; (2) the harness↔serve contract is a quality surface — a model that “fails agentic workloads” may just be crashing on parameters it was sent (kin: AI metrics that actually matter’s benchmark-vs-production drift; Hermes’ own --reasoning-effort handling).
- GSQ-RCO — the learned-grid quant family (DASLab): per-weight grids + Riemannian bit-allocation under exact size budget, standard GGUF for stock llama.cpp; sweet spot 3-5 bits near-lossless (sub-3-bit frontier belongs to runtime codebooks).
Strata in the field: the memory-budget reality check (2026-10-07)
Two field tweets, both captured: ItsmeAjayKV — “Whenever I run Qwen3.8-flash-next with Strata I cannot use anything else… not enough memory lol, see 58/60 GB” (41❤; the whole-RAM tax of the offload lane this note tracks) — answered by retardrutide: --resident-experts-gib 36 — pins 36 GB in mlock and mmaps the rest of the experts. The knob that turns “Strata eats the box” into “Strata coexists with the box”: resident experts get page-locked RAM (hot path, no swap-out), the long tail of MoE experts stays mmap’d (demand-paged, evictable under pressure — the OS reclaims pages when something else needs RAM). MoE makes this split natural: only a few experts fire per token, so pinning everything is waste; pin the hot set, mmap the tail. Ops note for the doombox lane: same physics as qmd-vs-llama-swap VRAM arbitration, one level up in the stack.
The free-compute pole: Qwen3.8-27B BF16 on Kaggle TPU (2026-10-07)
Oluwaphilemon1 (full analysis in the clipping): full BF16 Qwen3.8-27B on a free Kaggle TPU v5e-8 — no quantization at all, and the numbers are “wild”: ~130 tok/s decode, ~10,300 tok/s prefill, 262K native context (105K-token prompt in ~10 s; 225K in ~28 s). The prefill figure is the one that matters for coding agents — this note’s harness-latency lane says prefill, not decode, gates agent loops. ~540 tok/s aggregate across 8 concurrent streams — it’s a real serving layer, not a single-user stunt. Architecture: Kaggle TPU v5e-8 → Qwen3.8-27B BF16 → OpenAI-compatible endpoint → Claude Code / Codex CLI / OpenCode — existing agent tooling connects with no custom TPU app.
The honest limits, from the same source: quota-based free access, ~20 TPU-hours/week, sessions <9 hours — “not going to replace a machine you control 24/7.” Vault verdict: the interesting pole shift is what “local” means — free remote TPU as the ephemeral big-inference tier (full-precision + huge context bursts) beside the owned 3090 lane (quantized, always-on). Kin: GSQ-RCO (the precision axis this deliberately skips), the Bonsai/Strata serve-surgery entries (the owned-hardware answer on the same model family).
llmfit: the right-sizing calculator, now with measured numbers (2026-10-08)
AlexJonesax — llmfit 1.1.17 (tweet + llmfit (OSS) captured; 37.7k★ Rust): “hundreds of models & providers, one command to find what runs on your hardware” — auto-detects CPU/RAM/GPUs/VRAM/unified memory (CUDA, Apple Silicon, ROCm, OneAPI), projects memory footprints and tok/s across quantizations (GGUF, AWQ, GPTQ, EXL2), scores fit on quality/speed/fit/context; TUI (v1.1.17 adds mouse support), --memory-percent/--ram-percent flags, multi-GPU + MoE aware, REST API for pipelines, runtime-provider aware (Ollama, llama.cpp, MLX, Docker Model Runner, LM Studio). The feature that upgrades it from calculator to instrument: benchmark & share — measure real tok/s on your hardware, contribute back as a PR from the TUI; merged submissions ship measured ✓ numbers in the next release for anyone on identical hardware. USER TESTIMONIAL, verbatim: “Awesome TUI, I have used it.” For this note: the planning tool for the 3090/llama-swap lane — the VRAM-arbitration question (qmd vs llama-swap, Strata’s resident-experts) is exactly what llmfit’s --memory-percent budgets quantify before a download.
The DGX Spark datapoint: local Flash = frontier parity on everyday coding, 5–8× slower (2026-10-09)
TeksEdge (12❤) → the Reddit original (captured, 28.5 KB): one local DGX Spark (GB10, 121 GB unified, NVFP4 Flash via vLLM 0.30 + MTP spec-decode, ~30 tok/s long-gen) matched Claude Opus 5.5 11/11 on the everyday agent-coding tier — same score, ~7 min vs 36–59 min, “roughly 5–8× slower.” Hard tier: Opus 10/10 ($7.43 total) vs local best 7/10 — with the tell that one config was fully correct on 9/10 but timed out because it didn’t stop. The failure mode was over-verification, not inability (it fixed the bugs, then wrote its own stress-test harness and debugged that); “stop when tests pass” prompts didn’t help, lower reasoning effort did. Tuning lore that generalizes: Flash’s chat template defaults to reasoning_effort xhigh and overthinks — medium = 2.3× faster, zero lost passes; Qwen Code’s /effort doesn’t reach vLLM (wrong field: reasoning vs top-level reasoning_effort — set via extra_body); Qwen Code silently discards responses streaming >15 min and retries (QWEN_STREAM_MAX_LIFETIME_MS=3600000); harness choice moved hard-tier scores as much as settings. Real-world one-shot: 150-line spec → bilingual Next.js wedding microsite, 88 minutes, ~5.4k lines / 30 files, clean build, every functional requirement met — “eerily similar” to Antigravity 2.0’s output from the same spec. Takeaway: “for everyday coding, the local setup is good enough if you can live with the speed. For hard work, plan with a frontier model, build steps locally, and escalate the steps that stay stuck.” For the roadmap: the household llama-swap fleet’s ceiling is not capability but patience and effort-tuning — and the honest caveat thread’s own: Opus designed the hard tier, so 10/10 may be saturation-flattered (the eval-evaluates-the-evaluator seam again).
superfluid: the daemon that wants llama-swap’s job, with a scheduler and a WAL (2026-10-09)
superfluid (OSS) (basecompute, 6★ day-old Rust, Apache-2.0, pre-1.0; captured): “one serving daemon for local LLM inference — across runtimes and machines.” One scheduler + one durable session log (write-ahead — “a killed worker or a restarted server picks its sessions up where they were”) + one API surface (OpenAI/Anthropic/Ollama clients unchanged) in front of llama.cpp, MLX, and baseRT (each an isolated worker, nothing vendored; new runtimes = one adapter crate). The three claims, benchmarked on M5 Pro/Qwen3-4B vs each runtime’s own server: interactive preemption — a chat behind eight agents gets first token in 0.79 s vs 67 s (llama-server) / 64 s (Ollama) / 120 s (mlx_lm.server); shared-prefix prefill — 32 requests over one 5k prompt = ~1 prefill not ~8; serving-scenario suite 8/8 vs 3/5/1; throughput parity with llama-server at 1 and 8 concurrent. Also: superfluid launch claude --model ... (starts the server for a coding agent), and multi-machine: node join finds the head on the LAN, encrypted link, distributed serving. Positioning vs the household’s llama-swap: same “front of the engines” job, but llama-swap solves VRAM arbitration (one GPU, models swap) while superfluid solves interactivity and durability (many agents on shared models: preemption, prefill dedup, session WAL) — and it doesn’t do GPU-swap at all (persistent workers, includes Apple MLX). The doombox fit is the multi-agent future: when Hermes + coding agents + ad-hoc chats share one box, the 67s→0.79s preemption number is the feature that matters; today it’s a watch-item (pre-1.0, 6★).
Toilgate: the gateway-observability dashboard, with the 97%-cache tell (2026-10-10)
alexellisuk (Alex Ellis): “Jan’s driving TP2 GLM 5.3 Flash with Toilgate” — quoting jantytgat (“Look at this beauty chugging along! superterm toilgate localai”; both tweets captured + screenshot vision-transcribed): Toilgate v0.0.8 — a self-hosted terminal-aesthetic web dashboard sitting on an LLM gateway: per-user/per-day/per-hour token punch-cards, LIVE streaming connections (client badges SUPERTERM/OPENCODE), decode t/s, per-request KV-cache hit, model breakdowns, and a legend that reads the backend’s /metrics prefix-cache rate (“the Spark DS4F ring”). The numbers are the lesson: 536.1M tokens, 97% of input cached, contexts 23–39K, glm-5.3-flash at 32–65 t/s decode (41 t/s avg) + a Qwen3.6-27B-Base side-lane at 145 t/s — the workload profile of agent coding loops (huge near-identical contexts resent each turn), made visible. For the household lane: this is the missing observability piece between llama-swap (engine) and the AI-gateway lane (AI gateways — bifrost’s Prometheus/semantic-cache story vs toilgate’s human-facing punch-cards); when doombox runs multi-agent workloads, cache-hit share is the metric that says whether the serving stack is actually agent-shaped. Kin: the homelab monitoring stack’s VictoriaMetrics lane (the /metrics reader), maple’s OTLP-native agent investigations (the agent-era observability thesis).
nerkyor’s Coder390: EfficientThink receipts + the Q2-still-scores-90 quant table (2026-10-09)
xiax0603 (898❤/62k views; tweet captured, HF card 401-gated — benchmark slides vision-transcribed into the clipping): Qwen3.8-27B-Coder390-EfficientThink (nerkyor, Apache-2.0) — a community post-train that fixes the over-verification failure mode the DGX Spark benchmark documented: the base “keeps repeating its reasoning until the 94K token cap without producing a final answer”; Coder390 (SFT+RLOO on 3.9M code samples) cuts reasoning tokens 25–53% across P50/P70/P90 while raising scores — GPQA 89.4→89.9, MMLU 88.8→90.0, LCB 83→90 (94K-truncations on LCB: 13→3). The quant tables are the practical gold for this note’s GGUF lane: Q2 LynnStyle 12.4 GiB still scores LCB 90 (16G VRAM), Q4 18.3 GiB = 89 on 24G with built-in Q4 MTP, Q6_K+MTP single-stream 132 tok/s vs 53 without (2.5×); server-side NVFP4 W4A4 19.2 GiB w/ 976 tok/s (NInfer) and 1,146 tok/s peak (W4A4-W8A8 C16). Plus the honest operational detail: llama.cpp C8 concurrency cap, and when the KV pool fits only one 100K request, extra requests queue → HTTP 503 — the VRAM-vs-context budget this note keeps meeting, in one line. For the 3090 lane: Q4-K_M-class 27B with working MTP at ~18 GB leaves room for a translation side-lane; the EfficientThink approach (train the stop-behavior, not just the answer) is the durable idea — same fix as lowering reasoning_effort, but baked into weights.
Fine-tuned specialists beating frontier generalists: the Coinbase datapoint (2026-10-12)
Brian Armstrong (4.6k❤/453k views; thread captured; blog 403-walled, numbers preserved via secondary reporting — see clippings/coinbase-com-blog-part-one-fraud-agent-reported.md): Coinbase fine-tuned Qwen3.5-9B on proprietary internal data for fraud prevention and it outperformed frontier models on the specialized task — F1 +9.6%, end-to-end latency −55%, much lower cost. The org-side detail that matters: fine-tuning is employee-self-service via an internal-harness skill — /fine-tune-and-serve-llm I want to fine-tune Qwen for payments risk review. help me get started — and Armstrong projects “some backend services could be replaced entirely by this new architecture.”
Why this note carries it: it’s the production-scale confirmation of this note’s economics thesis — frontier generalists for breadth, small fine-tuned specialists where the data is proprietary and the task is narrow. A 9B model + internal data beats a frontier API on both quality AND latency AND cost — the trifecta local/small-model advocates predict but rarely get from a regulated fintech at scale. Pairs with the nerkyor Coder390 capture (community post-train fixing a base-model failure mode): the fine-tune is where proprietary advantage lives, and the harness skill makes it a normal employee action, not an ML-team project.
The doctrine, in one line: “Local for everything you can, frontier where it still wins” (2026-10-12)
Dr. Stephen J. Ronan (71❤/6.1k views; tweet + receipts repo captured, Apache-2.0): his home AI lab — two DGX Station GB300s with RTX PRO 6000 sidecars, three DGX Sparks, Threadripper server, all on solar and batteries — distilled into the sentence this note has been circling:
“Local for everything you can, frontier where it still wins.”
USER verdict at capture, verbatim: “That quote is golden.” It compresses the note’s whole position — the DGX Spark takeaway (“for everyday coding the local setup is good enough… for hard work, plan with a frontier model, build steps locally, escalate the steps that stay stuck”) and the Coinbase fine-tune datapoint (specialist local beats frontier where the task is narrow and the data is yours) into one routing rule. Note the honest tail: “where it still wins” — not “never,” not “always”: a moving boundary that each benchmark tier re-draws.
The repo is the receipts-first discipline this note’s numbers should always have: every number a row in results.jsonl, every row pointing at a scrubbed log, tools/validate.py checking both directions. Headline measurements (two Stations): DeepSeek-V4.1-Flash 5,774 tok/s at 64 users / ~430 single; sidecar expert-offload 2.7-3.25× at 16-32 users; DS V4-Pro 1.6T at 83-95 tok/s single; speculative decoding “measured honestly: 1.77× on real text, slower on random tokens.”