Small Vision Language Models

The under-10B class of image-capable models — the ones a single consumer card (the 3090 reference) can run for generation, editing, or understanding. The local-AI cluster had text (Qwen model family) and STT (Speech to text inference); this is the vision shelf. “Small” is doing double duty: small enough to fit, and increasingly small enough to win — the 2026 releases punch far above their parameter class.

Text-to-image / design (generation side)

• Ming-Image-0.1-design (AntLingAGI; Comfy-Org weights captured) — 6B text-to-image for UI, infographics, posters, text-rich design — the design-tool niche, not art. @ItsmeAjayKV hands-on on a 3090: int8 weights, ~25 steps (12 recommended at higher precision), images within a minute, “really good at design, infographs and text rendering” — text rendering being the historic failure mode of image models. First-party ComfyUI packaging.

• Qwen-Image-2.1 (model card captured) — 7B visual component (32 single-stream DiT layers), unified generation and editing: native RGBA transparency, up to 10 reference images, circle/painted/mask local edits, identity preservation for people and products; mixed-granularity attention + prefix KV-cache reuse for efficiency. Per the captured tweet thread: positioned as “just the appetizer” — Qwen-Image-3.1 is the announced next step, so the family line is active.

• PixelUMM (NVIDIA nv-tlabs, 131★; HF paper 2609.38597, weights on nvidia/PixelUMM; via @CongWei1230 — first author reply) — the architecture-heresy entry: encoder-free unified understanding + generation, no VAE and no ViT. Reads AND writes raw pixels with a single decoder-only Transformer: images = 16×16 patches, videos = 4-frame tubes; Qwen3-8B backbone with separate understanding/generation experts sharing one self-attention across text, clean pixels, and noisy pixels (diffusion-style, in pixel space). Two checkpoints: S8-F22-R05 (paper default) and S8-F18-R01 (extra 480p/720p fine-tune, better T2V); T2V gated behind NVIDIA Cosmos guardrails (on by default). Why it matters here: the Qwen-8B-class backbone keeps it in “small enough to matter to a 3090” conversations, and it’s the strongest datapoint yet for the pixel-space convergence thesis — if raw-pixel models match latent-space ones, the VAE/ViT preprocessing stack (and its training cost) becomes optional.

Reading list / growing

• (resolved — lift (OSS): 9B document→JSON extraction model from datalab; the understanding side arrives) lift (datalab-to, 1.2k★, Apache-2.0; @akshay_pachaar announcement) — open-weights 9B that takes a PDF/image + a JSON schema and returns a matching JSON object in one step: OCR + LLM + JSON-repair chain collapsed into a single pass that reads every page at once (values spanning pages still resolve). 90.2% field accuracy across 11,000 fields, ~matches Gemini Flash 3.5 at 3× the speed; built not to hallucinate — returns null when a field is missing instead of inventing it. Self-hostable via vLLM or HuggingFace (weights on HF), pip install lift-pdf CLI, Streamlit Schema Studio. License note: Apache-2.0 source, but commercial self-hosting needs a license — the open-weights-not-open-product pattern. Fits the oncall/invoice-class extraction lanes (and the witcheer invoice-facts benchmark family).

• (still open: small VLMs for OCR/document/screen understanding (GLiNER-adjacent multimodal, the Fine-tuning GLiNER ticket-classification extension); Molmo/SmolVLM-class; where Strata’s vision GGUF (mmproj pattern from the JetBrains blend) fits)

• (open: does Ming’s design-niche beat generic models on UI work, or does Qwen-Image-2.1’s editing versatility win the slot? — hands-on A/B on the 3090)

Related: Small language models (the text-model sibling) · Local LLM inference · Inference engines · Qwen model family · Speech to text inference (the other modality shelf)