brain

Small language models

Properties7
statusseedling
typeconcept
created2026-10-05
updated2026-10-08
tagssmall-language-models, slm, local-ai, inference, models, edge
aliasesSmall language models, SLMs, small models, small LMs
public

4 min read

Small language models

The under-~10B class that runs on single consumer hardware (the 3090 reference, Apple Silicon, or smaller) — the text-model sibling of Small Vision Language Models and the model shelf of Local LLM inference. “Small” is again doing double duty: small enough to fit, and increasingly small enough to win a job — the 2026 pattern being small models trained/distilled for one job done at frontier-adjacent quality, not small models pretending to be general.

The roster (vault-grounded)

  • Gemma4-12B-Q4KXL/-mtp2 (Google; in the doombox llama-swap lineup) — the pinned cheap-slot candidate from the portal-affordable analysis: describers/labelers/titles. MTP heads for speculative-style speed.
  • Qwen family small tiers — Qwen3.5-based Index-Translate (OSS) 2B/9B (translation lane); Qwen3-4B-SafeRL (safety-aligned via guard-model RL feedback); Qwen3-8B as PixelUMM’s backbone (understanding+generation in pixel space); lift (OSS) 9B (document→JSON extraction).
  • Qwen3Guard 0.6B/4B/8B — the guardrail lane: token-level Stream moderation during generation.
  • basal-1.5 (KinasRemek) — “first Polish system-one classifier” class; decision-model lane (Decision models).
  • GLiNER2-class — small open NER/labeling models (ticket classification, Fine-tuning GLiNER).
  • Phi-class / SmolLM-class — the classic tiny-general lane (watch, not vault-critical).
  • NInfer + Qwen3.8-27B on one 3090 (omp-ninfer, 42★) — restart-resumable durable local inference at the 27B tier; the upper edge of “small” on 24 GB.

Why small models are winning jobs in 2026

  1. Slot pinning — the witcheer/portal-affordable principle: pin the cheap slots (describers, labelers, triage, compression candidates), never the quality gates. A 12B at 0.002/call for a job both do well.
  2. Verifiability asymmetry — for jobs with checkable outputs (JSON schema, glossary translation, classification labels, guardrail verdicts), small-model errors are catchable, so frontier quality isn’t needed — CLIs for agents’s idempotency/structured-errors logic applied to model choice.
  3. Speed/latency lanes — token-level moderation (Qwen3Guard (OSS) Stream), real-time translation, speculative drafting (DFlash2 MTP/block-diffusion) — small models as sidecars to big ones, not replacements.
  4. Privacy/data residency — enterprises running locally (mfranz_on’s guardrail tweet); nothing leaves the box.
  5. Distillation economics — frontier model as teacher, small model as deployed artifact (SafeRL’s guard-feedback loop is this pattern turned recursive).

The design pattern

Small models compose into pipelines with roles: drafter (speculative), guard (moderation), extractor (schema), translator (glossary), classifier (labels) — each pinned, each with its own acceptance test (witcheer’s 10-facts retention test, lift’s 90.2% field accuracy, Guard’s benchmark suite). The failure mode is the opposite slot: asking one small model to be general (the 2023 mistake).

Related: Local LLM inference · Small Vision Language Models · DFlash2 (speculative drafting) · Decision models · Qwen3Guard (OSS) · lift (OSS) · Index-Translate (OSS)

  • Embedding models — retrieval as the lane where small models win first (per-token cheapest work); EmbeddingGemma 2: 740M multimodal in 0.5 GB RAM.

Vela 2.0 0.3B: the routing/classifier job, now on-device (2026-10-08)

Alex_tra_memory (215❤) quoting XunzhuoLiu (vLLM team): Vela 2.0 — open Foundation Routing Models bringing span-level decisions to System One; the demo that matters: Vela 2.0 0.3B running on-device on an M5 Pro MacBook via Core ML — 1,000 chatbot replies screened in ~11 s (~90 msg/s): jailbreaks + harmful blocked, PII redacted, the rest routed to a team. 99.4% of ops on the Neural Engine. Artifacts: FluidInference/vela-2.0-0.3b-coreml + FluidUse (OSS) (247★; Apple-silicon local computer-use stack — typed decisions with laya, form-filling via CUA-S1-FORMS, driven through the macOS Accessibility API). For this note’s thesis: the sharpest “small enough to win a job” specimen yet — routing/safety-screening is exactly the job an on-device 0.3B beats an API round-trip at, and it composes with the vault’s PII redaction and Agent sandbox lanes (the interception layer, cheap enough to run per-message). Kin: OrcaRouter (quant-fidelity reporting kin in Local LLM inference), Weight poisoning (the defensive small-model stack).

Graph View

Backlinks