Local AI stack

The running local-AI system on the homelab’s single GPU box (Ryen 5800X, RTX 3090 24 GB, 32 GB RAM) — every number below verified against live state on 2026-10-12. The concepts and engine trade-offs live in Local LLM inference and Inference engines; this note is the machine as it actually runs.

Architecture

One GPU, one resident model, everything else is plumbing around that constraint.

flowchart LR
    subgraph clients ["Clients"]
        HERMES["Hermes Agent profile"]
        LITELLM["LiteLLM router :4000<br/>(local + cloud fallback)"]
        APPS["LAN apps"]
    end
    SWAP["llama-swap :8080<br/>single OpenAI-compatible port,<br/>hot-swap proxy"]
    SERVER["llama-server<br/>ONE resident model<br/>(upstream ports 8001+)"]
    GPU["RTX 3090 24 GB"]
    TTS["OmniVoice TTS<br/>~0.4 GB lean mode"]
    STT["voiceapi STT :5092<br/>Parakeet, CPU-only"]
    MON["vmagent + Grafana"]
    HERMES -->|"chat + vision"| SWAP
    LITELLM -->|"chat"| SWAP
    APPS -->|"chat"| SWAP
    SWAP -->|"spawn / SIGTERM"| SERVER
    SERVER --- GPU
    TTS --- GPU
    HERMES -->|"voice messages"| STT
    SWAP -.->|"/metrics llamaswap_*"| MON
    SERVER -.->|"/metrics llamacpp:*"| MON

The layers:

• llama-swap v216 (llama-swap (OSS), Go, MIT) — a systemd user service listening on :8080, presenting all registered models behind one OpenAI-compatible endpoint. --watch-config re-reads the YAML every 2 s, so profile edits hot-reload without a restart. Upstream llama-server ports are assigned alphabetically from startPort: 8001. A /metrics endpoint (prefix llamaswap_*) plus each server’s own llamacpp:* metrics feed vmagent/Grafana.
• llama.cpp llama-server (CUDA build 10450) — the engine. Full GPU offload (-ngl 99), flash attention, KV cache quantized to q4_0 on both keys and values, single slot (-np 1), --jinja chat templates, --metrics --slots for per-slot Prometheus telemetry. DFlash2 profiles run a PR-pinned server binary that carries the drafter’s p_min acceptance filter.
• Model shelf — 7 families, 61 profiles, ~118 GB of GGUF on disk. Quants are Q4_K_M-class (UD-Q4_K_XL / Q4_K_IQ4_XS) from Unsloth/bartowski/orcarouter, with small Q2_K drafters for the block-diffusion speculation lane.

Model families

Family	What it is	Profiles	Extras
Gemma4-12B	QAT Q4_K_XL, natively multimodal	8	vision on every profile; the coexistence-safe default class
Qwen3.6-27B	dense workhorse	8	—
Qwen3.6-35B-A3B	MoE (35B total / 3B active)	8	—
Qwen3.8-27B	current daily driver, multimodal	14	widest variant set: dflash2, mtp6+ngram stacks
Ornstein3.6-27B	Gestalt Labs NSC-ACE-SABER tune	9	ngram self-speculation
Orca38-27B	orcarouter uncensored Qwen3.8 finetune	8	vision
Ornith-1.5-35B-A3B	AtomicChat MoE	6	dflash drafter lane

Profile grammar: {Family}-{quant}, -think suffix adds --reasoning on; -mtp{n} adds MTP speculative decoding at depth n (2/3/4/6); -ngram stacks n-gram self-speculation on top; -dflash pairs a Q2_K block-diffusion drafter (DFlash2). Context follows the speculation mode: 262K plain, 131K with MTP (the draft head costs ~2 GB of KV headroom), 245K for dflash2.

The variant matrix for one family looks like this — same weights, six ways to spend the VRAM:

flowchart TD
    BASE["Qwen38-27B Q4_K_M<br/>262K ctx, no speculation"] --> THINK["-think<br/>+ reasoning mode"]
    BASE --> MTP["-mtp3<br/>MTP depth 3, 131K ctx"]
    MTP --> MTPT["-mtp3-think"]
    MTP --> NGRAM["-mtp3-ngram<br/>+ n-gram self-speculation"]
    NGRAM --> NGRAMT["-mtp3-ngram-think"]
    BASE --> DFLASH["-dflash2<br/>Q2_K block-diffusion drafter,<br/>245K ctx, +32-46% tok/s"]

Basic config

Shared flags on every profile: -b 4096 -ub 1024 (batch sizes tuned to survive the single-3090 cliff), -ngl 99, flash attention on, --cache-type-k/v q4_0, -np 1, --jinja, metrics. Qwen-family profiles use a community fixed chat template (froggeric v22) with deepseek-style reasoning formatting — it fixes tool-call fidelity that the stock GGUF template gets wrong, and exposes a per-request reasoning_effort knob so thinking volume can be dialed without a profile swap.

Two operational defaults worth knowing:

• Startup preload — the service loads a 12B vision profile at boot, the only model class that coexists with the TTS service in both its lean and full modes.
• TTL disabled (globalTTL: 0) — a resident model stays loaded until the next swap or a manual unload; idle time does not free VRAM.

Hot-swap semantics

The resident model is whatever was requested last through :8080 — not the preload, and not what /v1/models says (that endpoint lists all 61 profiles and proves nothing about residency; the actual llama-server process args are the truth).

sequenceDiagram
    participant C as Client
    participant P as llama-swap :8080
    participant A as llama-server (model A)
    participant B as llama-server (model B)
    C->>P: POST /v1/chat/completions (model=B)
    Note over P: requested model ≠ resident model → swap
    P-->>A: SIGTERM (~1.3 s, VRAM freed, KV cache lost)
    P->>B: spawn + load weights + preallocate KV<br/>(5 s for 12B, 15-30 s for 27B)
    B-->>P: health check OK
    P->>B: forward request
    B-->>P: stream tokens
    P-->>C: streamed response
    Note over B: stays resident until next swap (TTL off)

Measured behaviors: unload takes ~1.3 s (SIGTERM, kernel reclaims VRAM immediately); cold-load-to-first-token is ~5 s for a 12B and 15–30 s for a 27B. The KV cache dies with the process — llama-swap has no session persistence, so a swap means every conversation re-prefills.

VRAM budget

The 24 GB card is the whole design space. Live snapshot (2026-10-12, dflash2 27B resident):

pie showData title RTX 3090 VRAM, measured 2026-10-12 (GB)
    "llama-server (27B Q4_K_M + 245K q4_0 KV + drafter)" : 22.0
    "OmniVoice TTS (lean mode)" : 0.35
    "free" : 1.6

Envelope rules from measurement: a 27B at Q4_K_M with 131K q4_0 KV and MTP depth 3 runs at ~21.9 GB — 27B and the TTS’s full mode do not coexist, 12B and anything do. Worry above ~23 GB, not ~20 GB. Speculation is how this stack buys speed inside that envelope: MTP acceptance runs 60–90% when healthy, DFlash2 adds +32–46% tokens/s at 32–131K depth, and n-gram self-speculation is neutral at short context but +8–13% at 131K.

The speech lane

• STT — voiceapi, an OpenAI-compatible /v1/audio/transcriptions endpoint running Parakeet TDT 0.6B v3 (INT8 ONNX) with Silero VAD chunking, CPU-only by design: the GPU belongs to llama-swap. Speech to text inference has the full comparison lane.
• TTS — OmniVoice (k2-fsa) on the GPU in lean mode (~350 MB), sized to coexist with a resident 27B.
• The glue — a Hermes Agent profile uses the stack end-to-end: chat/vision via Qwen38-27B-dflash2-think, voice-message transcription via the local STT endpoint. LiteLLM (:4000) fronts llama-swap for everything local and falls back to a cloud GLM endpoint — “local for everything you can, frontier where it still wins”.

Operating notes

Verified behaviors that bite if unknown:

• Config edits hot-reload via --watch-config; adding a profile shifts the alphabetically-assigned ports of every profile after it, which silently staleates port-based monitoring labels until regenerated.
• The unload API (POST /api/models/unload) frees all VRAM in ~1.3 s — the escape hatch before launching anything else that wants the GPU.
• Recompiled server binaries are picked up on the next model load, not by the currently resident process.
• Vision needs the --mmproj projector flag per profile; it boots fine without it and then silently ignores images. Think-variants return empty content for image input — vision tasks go to non-think profiles.

Related: Local LLM inference · Inference engines · llama-swap (OSS) · DFlash2 · Qwen model family · Speech to text inference · Hermes Agent · Homelabs