Inference engines

The execution layer of Local LLM inference — the software that turns weights + tokens into throughput. The choice axes: single-user latency vs multi-tenant throughput; consumer GPU vs datacenter card; GGUF quantization ecosystem vs raw performance; and now, speculative/heterogeneous tricks (the 2026 frontier).

llama.cpp

The incumbent of local inference (ggml/llama.cpp): C/C++, runs everywhere from phones to 3090s, the GGUF format is the local-model lingua franca. CPU+GPU flexible offload (-ngl), the richest quantization zoo (K-quants, imatrix, per-tensor), OpenAI-compatible llama-server, speculative decoding (greedy + draft models, incl. the DFlash2 block-diffusion drafter via PR #27342). The engine the doombox actually runs — llama-swap (OSS) hot-swaps llama-server instances; the Hermes iris profile rides on it. Strength: portability + quant ecosystem + single-stream latency. Limits: no continuous batching story at the server level, single-GPU focus.

llama.cpp forks

The fork ecosystem is where quantization and performance experimentation happens before (if ever) landing upstream. The ~/projects/llamacpp-tools scripts build each one locally (CUDA, install to ~/.local/bin, own clone + own build dir per fork so nothing touches the canonical ~/git/llama.cpp):

• ik_llama.cpp (ikawrakow) — the long-running performance fork (since 2024-06, diverged from upstream): additional SOTA quantization types, better CPU and hybrid GPU/CPU performance, first-class Bitnet support, DeepSeek-side MLA/FlashMLA + fused MoE ops, tensor overrides for hybrid inference, row-interleaved quant packing. Wiki benchmarks show 1.75-4x prompt-processing gains on i-quants vs mainline; stays current (Qwen3.8-Flash-Next CUDA work merged). Build: compile-ik-llamacpp.sh.
• llama.cpp-dflash2 — not a fork but a pinned checkout of **PR 27342 (the DFlash2 speculative-decoding PR) as its own clone; the runtime behind the iris profile’s Qwen38-27B-dflash2-think. The pattern: track a PR head as a branch when the feature predates upstream merge. Build: compile-dflash2-llamacpp.sh.
• llama-cpp-turboquant (TheTom) — TurboQuant fork; binaries installed with a tq- prefix so multiple engines coexist in ~/.local/bin. Build: compile-tq-llamacpp.sh.
• buun-llama-cpp (spiritbuun) — spiritbuun’s fork, tracked for its quant/perf experiments. Build: compile-buun-llamacpp.sh. New (2026-09-28): --resume — persists warm KV, slots, checkpoints, drafter states, and host prompt cache to disk on exit and reloads on start, saved per model arch; --slot-save-path uses the new format so existing setups (llama-swap) get it automatically; --resume-path for a custom dir, --resume-no-host-cache for slots-only. Stated mission: “kill cold prefill”, plus prompt-cache improvements for static KV codecs and dynamic VBR. Picked as the local benchmark case for the KV-cache prefix-sharing stretch goal in AI engineering stretch goals (TTFT cold vs --resume vs --resume-no-host-cache). Full flag diff vs upstream (from local source, merge-base 2026-09-28, buun 1,529 commits ahead): 2026-09-29 buun-llama flag diff — the --resume persistence family, a complete dynamic-VBR KV subsystem (--vbr-*), an integrated DFlash/MTP/diffusion speculation block (--spec-*, --dflash-max-slots), adaptive MoE expert caching (--moe-cache = cafe-llama’s territory but automatic), and --mmproj-gpu-swap for vision+speculation coexistence.
• DJLougen/llama.cpp — the rys-qwen35 fork (Qwen3.5-era tweaks). Build: compile-djl-llamacpp.sh.
• llama.cpp-mirai-s (alesha-pro) — the 2.4-bit frontier (thread captured): ported Mirai Labs’ bitshift-trellis compression of Qwen3.8-27B (2-3 bits/weight, runtime-computed codebook) by adding 4 ggml types + 2 ops and porting their CUDA kernels — bit-exact with Mirai’s vLLM plugin (greedy 8/10 token-for-token, 2 splits on 0.0007-logprob near-ties). Runs the 27B with 128K context in 12 GB (q4 KV) at 40 tok/s decode on a 3090; AppWorld agent scores at 2.4 bits beat bf16 (57.9% vs 55.8%, hardest-34: 52.9% vs 41.2%) where other ~2-bit quants collapse (GSQ-RCO IQ2_XS 33.3%). The generic fixes that came out of it (GQA 6:1 KV-head packing, chunked DeltaNet prefill, tokenizer-split cache-match bug that forced 30-54K re-prefills) are plain llama.cpp code, upstreamable. Not in llamacpp-tools yet — candidate fork for the build-script treatment.
• beellama.cpp (OSS) (Anbeeld, 1.1k★, MIT) — the KV-cache-focused fork: KVarN (variance-normalized KV quantization — better quality per bit than plain q-types, independent K/V widths kvarn2-kvarn8), a precision tail (recent N tokens kept F16 under a single global softmax, no full-cache materialization), low-bit cache types, and speculation-side upgrades directly relevant to the DFlash2 setup: KVarN for drafter caches (--spec-draft-type-k/v), independent draft micro-batch sizing (-ubd, saves gigabytes with DFlash), adaptive draft-max for DFlash (runtime horizon adjustment vs a no-spec baseline instead of fixed --spec-draft-n-max), plus reasoning-loop protection. Their fidelity table shows kvarn5/kvarn5 matching q8_0 quality at ~36% of the BF16 cache size. The most directly applicable fork to the iris profile’s 245k-context q4_0 KV setup — half the KV memory at equal quality is another 100k+ of context on the same card. Candidate for the build-script treatment.
• cafe-llama.cpp (OSS) (quimmedes, 49★, MIT) — the MoE-offloading fork: granular flags to place MoE expert weights in pinned host RAM (-hmoe, zero-copy async DMA over PCIe), CPU RAM (-cmoe), or streamed from SSD via mmap (-ssd — only the experts a token actually selects page into RAM, the rest stay on disk), with first-N-layers variants (-nhmoe/-ncmoe/-nssd) for hybrid tiering, plus -hmoed for drafter-side MoE offload — attention, KV cache and routers stay on the GPU. Explicitly targets Qwen3.8-Flash-Next and DeepSeek. Same design territory as Strata’s tiered offload, but as llama.cpp flags on any MoE GGUF rather than a bespoke engine — the open-source-community answer to “my 3090 + 32 GB RAM wants the 125B too”. Candidate for the build-script treatment.

Fork rules from the tools project: one script per fork, own clone and build dir (never build inside the canonical checkout), prefix or isolate binaries, auto-clone + always-pull-latest before building. The same discipline as Golden repo dependency pinning — a fork is a dependency, not a toy.

vLLM

HyperQwen (syv-ai/qwen38-27b-rtx3090, 1.8k★, Apache-2.0, captured 2026-10-03; note OSS/HyperQwen (OSS).md) — the 3090-serving specialist, directly relevant to this host’s card: Qwen3.8-27B on a single 24 GB card with vLLM patches — 127 tok/s single-stream, ~1,035 tok/s aggregate at 64 concurrent, 150k context (262k with KVarN). It is “really a patch series against a pinned vLLM plus a model-preparation pipeline — requantized heads, a calibrated draft vocabulary, speculation that drafts out of the prompt”. Latest update (r/LocalLLaMA 1vtup5s, captured): DFlash2 verify blocks decoupled from draft length (DFLASH_TOKENS=15: 15/16 tokens accepted when output quotes the prompt → 382 tps reproducing a 25k-token doc, +9% ordinary chat), int8 KV past 64k (138k tokens), and the honest corrections — KVarN decode tax is 2.13× single-user at 112k under speculation (documented as ~20% from batch-config measurements), +53% only where the model reproduces context, “a mode for RAG front-ends and coding assistants applying edits, not a default”. Strata’s credits list it as an idea source. The lookup-augmented drafting (draft from the request’s own context) is the same prefix-reuse family as buun-llama --resume and SGLang RadixAttention — the AI engineering stretch goals KV-cache plane has a third local benchmark case.

The throughput standard (vllm-project): PagedAttention (KV cache as OS-style pages — no fragmentation, prefix sharing across requests), continuous batching, tensor parallelism across GPUs. The default for serving many concurrent requests on datacenter cards; every managed API quietly runs it or something shaped like it. Strength: throughput per GPU at concurrency. Limits: CUDA-centric, heavier ops story, quantization support trails llama.cpp’s zoo (though improving).

SGLang

The structured-generation specialist (sgl-project): RadixAttention — automatic KV-cache reuse across requests sharing prefixes (a tree of cached prefixes; agentic workloads with shared system prompts hit hard), plus frontend DSL for constrained decoding (structured outputs, JSON schemas). Often beats vLLM on agent-shaped traffic for exactly that reason. The Fastino GLiNER docs assume it. Strength: prefix-heavy workloads, structured output at speed. Limits: same datacenter assumptions as vLLM.

ExLlamaV3

exllamav3 (OSS) (turboderp, 1.5k★, MIT, Python) — the consumer-GPU quant specialist: EXL3 quantization based on QTIP (trellis-coded — the same family of trick as Mirai’s bitshift trellis: squeeze below 3 bpw with quality that conventional quants can’t hold at that size), 2-8 bit KV cache quantization, flexible tensor-parallel and expert-parallel across consumer cards, CPU offload with AVX2/AVX512 for big MoE, continuous batching + speculative decoding. Server story via TabbyAPI (OpenAI-compatible). The lineage: EXL2 (v2) was the quality-per-GB quant format before imatrix GGUFs caught up; v3 pushes the frontier further down the bpw curve. Position vs the rest: llama.cpp owns the format ecosystem (GGUF everywhere), ExLlamaV3 owns the low-bpw quality ceiling — its .exl3 quants consistently top low-bpw comparisons where GGUF i-quants fall apart.

lucebox

lucebox (OSS) (Luce-Org, 2.9k★, Apache-2.0, C++) — speculative inference for heterogeneous machines and consumer GPUs: per-model-and-hardware custom kernels, speculative prefill + decoding. The numbers that matter to a 3090 owner: PFlash + KVFlash on an RTX 3090 at 256K context — 6.1x prefill (411 s → 67.3 s), 152.3 tok/s decode with an 8K pool; Luce Spark ~100 tok/s in 14.6 GiB; megakernel 413 tok/s at 1.87 tok/J on a 3090. Also the DFlash2 host with the fastest published Qwen3.8-27B single-GPU number (208 tok/s on an R9700). The engine to watch for the homelab: it treats consumer hardware as a first-class target, not a fallback.

Strata

Strata (OSS) (Niko1221, C++) — not a general engine but the offload specialist: Qwen3.8-Flash-Next (125B/6B-active) on one gaming PC by tiering GPU (dense + hot experts) / RAM+CPU (remaining experts) / SSD (51B n-gram table), with MTP speculation at 1.6-1.8x lossless. Where lucebox optimizes fast, Strata optimizes fits: it answers “can this machine run it at all?” for MoE models the other engines won’t touch on consumer hardware.

Magnitude

magnitude (OSS) (YC S25, 5.8k★, Apache-2.0, Rust) — the self-optimizing, agents-first local engine: kernels written with flexible parameters that are tuned on your actual device before first run (on-device autotuning — broad compatibility at hardware-specific performance), popular-families-only kernel depth, dynamic memory allocation (reserves weights-only up front; heap grows with sessions, frees on agent exit — the computer stays usable), hybrid paged attention (SGLang-style shared prefix caches with memory-adjacency placement so single-session speed doesn’t pay for it). Launch numbers vs llama.cpp (Qwen 3.6 35B A3B 4-bit, 64k ctx, no spec decode): Metal M4 Pro +92% decode (30→57 tok/s), DGX Spark CUDA +19% decode/+23% prefill, ~28% less per-agent memory both. Catalog ships with per-model drafters (DFlash/DSpark/DFlash2) and TurboQuant-inspired KV (8-bit K / 4-bit V — >2x KV memory cut, RULER-verified long-context retrieval); desktop app auto-runs models on demand for Pi/OpenCode/Hermes/Codex. HN thread’s counterweights (captured): MLX beats it on Apple Silicon in user benchmarks (161 vs 175 tok/s decode on M5 Max) — the llama.cpp comparison is the low bar on Mac; the real differentiators are agent-shaped: KV quant at 100-200k contexts, multi-session memory, prefix reuse. Roadmap: expert streaming (models bigger than VRAM), kernel compiler, multi-device layout solving.

Learning resources

• LearnInference (learn-inference.com, captured 2026-09-29) — free interactive companion to Inference Engineering by Philip Kiely: “everything after training… where the bill actually lands” — attention kernels through GPU procurement across three clouds. The pedagogical move is simulators: “I built simulators for the parts you grasp faster by turning a dial than by reading a paragraph” (e.g. the TTFT-vs-tokens/s slider from ch. 1 — a response that starts instantly and trickles feels faster than one that pauses and dumps). Chapter-structured, full glossary. The right on-ramp before the dense sections of this note.

How they slice

• Who’s it for: llama.cpp = everyone/everything portable; vLLM = many-request serving; SGLang = agent/prefix-heavy serving; ExLlamaV3 = lowest-bpw quality on consumer cards; lucebox = consumer-GPU performance; Strata = consumer-GPU capacity for giant MoE; Magnitude = agents-on-a-desktop (multi-session memory + autotuned kernels)
• KV cache: vLLM paged / SGLang radix-tree reuse / llama.cpp straightforward; lucebox KVFlash pools it; Magnitude 8-bit-K/4-bit-V quant + hybrid paged (prefix sharing without single-session loss)
• Speculation: llama.cpp draft models + DFlash2; lucebox speculative prefill+decode; Strata MTP; vLLM/SGLang have it but it’s not their identity; Magnitude catalog-assigned drafters. First-party DFlash drafters now published by model vendors themselves — ornith-ai for the whole Ornith-1.5 family (2.54× lossless claimed), the pattern DFlash2 started generalizing
• Local reality check: the box runs llama.cpp via llama-swap today; lucebox’s 3090 numbers make it the first candidate worth an A/B when the roadmap next revisits serving; Magnitude is the agents-first alternative when multi-session + long-context is the workload (its DGX Spark class ≈ nothing we own, but the memory model fits doombox + concurrent Hermes sessions); vLLM/SGLang become relevant the day something serves concurrent users (the gateway? GLiNER batch jobs?)

Related: Local LLM inference (the layer above) · llama-swap (OSS) · DFlash2 · Qwen model family · Strata (OSS) · lucebox (OSS) · magnitude (OSS)

Serving on Kubernetes (the ops-shaped tutorial)

jubril.xyz — Inference on Kubernetes (26.6 KB, captured; via @s1ntaxe770r — “the cover is Gwen cuz it rhymes with Qwen and I’m funny”) — the rare inference tutorial written for operators: what makes an inference workload different from a standard app (tokenization → prefill [single parallel forward pass, builds the KV cache] → decode [cyclic per-token dependency, cannot be parallelized — why speculative decoding exists, see DFlash2), then the full k8s lane on CPU-only (Civo k3s, 16 vCPU/32 GiB, no GPUs — “thanks to Mr Altman and friends, GPUs are expensive”): CPU-instruction checks inside pods (VMs expose only some host capabilities), model choice for the budget, download-before-start init containers, vLLM config, and the operator-grade cold-start honesty — startup probe allowing ~20 minutes for /health (weight loading ≠ container running), readiness gating the Service. Capped with Agentgateway (Rust, Gateway API CRDs): a “model-aware” layer over Kubernetes Gateway API speaking A2A + MCP, for rate limiting and per-model metrics. The piece that generalizes: inference workloads need serving discipline (probes that understand weight-loading, init containers for artifacts, gateways that understand tokens), not just a Deployment.

KServe from Zero to Production (Medium, Asmaa Chebba, 57.8 KB captured — Cloudflare-walled, fetched via reader; via @KubeBuilders, the KServe org account) — the operator-grade counterpart to the jubril.xyz piece: assumes Kubernetes deeply, teaches everything KServe adds on top. Part I: InferenceService reconciliation flow and per-component resources, Standard vs. Knative deployment modes (raw Deployments vs. scale-to-zero + request-driven), Gateway API vs. Ingress networking, Multi-Model Serving (packing many models on one predictor to dodge the one-pod-per-model GPU tax), RBAC scoped to KServe CRDs + endpoint authn/authz, canary rollouts with metrics-based promotion. Part II builds a Kind cluster (Calico + Gateway API) with a classic ML model and a small LLM; Part III is the production-readiness ladder — custom predictor from scratch, real autoscaling under load, observability, Gateway API Inference Extension for LLM fleets, LLMInferenceService end-to-end. Together with the jubril piece the vault now holds both halves: workload discipline (probes/init containers/weight-loading) and fleet discipline (reconciliation, multi-model packing, canary, scale-to-zero).