Embedding models

The retrieval half of the RACK stack — models whose output is a vector, not text. One job: map content into a space where similarity means relatedness, at a price/latency an index can afford.

The space

• Text embedders (the classic lane): e5/BGE/multilingual-e5, OpenAI/Voyage/Cohere hosted — what qmd embeds this vault with today.
• Multimodal embedders (the new lane): one shared space across text + code + image + audio + video — EmbeddingGemma 2 (Google DeepMind + Unsloth, 1,677❤ / 73k views, both captured): first natively multimodal open model for on-device embeddings — 740M params (270M text + 170M vision + 300M audio), Apache 2.0, runs in 0.5 GB RAM, GGUFs on HF, runnable/trainable via Unsloth. The significance: CLIP-class unified retrieval no longer needs a service — it fits inside the harness/agent process (compare Codemode’s harness-side engines, MCP reconsidered).
• Special-purpose embedders: code-search models, rerankers (cross-encoders — technically scorers, kin of the family).

Design axes that matter

1. Dimensions vs. recall — bigger vectors buy nuance, cost index memory and search time (HNSW graphs scale with d).
2. Matryoshka (MRL) embeddings — trained to stay useful when truncated: one model, many sizes; store big, query small.
3. Context length — long-doc embedders avoid chunking damage.
4. Unified vs. per-modality — shared space = cross-modal search for free (“find the slide that says X”); per-modality = better in-domain precision.
5. Local vs. hosted — privacy/latency/cost; the 2026 story is local catching up (EmbeddingGemma 2 at 0.5 GB; llama-swap-servable).

Vault kin

• Small language models — same “one job at frontier-adjacent quality” thesis, applied to retrieval; embedding is the cheapest per-token work, so small models win here first.
• Agent knowledge systems — CorpusMap’s result (entity linking beat LLM-wiki and raw search) puts embedders in their place: part of hybrid retrieval (HNSW + BM25 + RRF), not the whole answer.
• CortexDB (both) — .ai: OpenAI embeddings by default, fails closed without a key; Go namesake: lexical mode with no embedding model at all — the two ends of the embedder-optional spectrum.
• qmd itself — hybrid FTS+vector; embed runs CPU-only on this host (Agent knowledge systems tending loop).
• Local LLM inference — Strata’s 51B n-gram embedding lookup on SSD: embeddings ≠ always neural; statistical lookups still compete at the low end.

Practice (homelab)

• EmbeddingGemma 2 GGUF is a natural llama-swap/qmd-side experiment: 0.5 GB RAM means it coexists with the 3090’s LLM slots.
• Before swapping embedders: re-embed is O(corpus) — measure on a slice, compare recall@k on known queries, then commit (the eval discipline from LLM evals applies to retrieval too).
• Lexical-first (BM25/FTS5) remains the baseline that any vector layer must beat — CorpusMap and the Go cortexdb both say so.

The homelab research: the language-dimension axis (2026-09)

The deepest local dive predates this note — the homelab local-lm-roadmap §2 (“Embedding model comparison”, private repo) sized candidates for mixed Polish/English RAG, and it adds the axis the public debate ignores: a monolingual-quality model for the primary language beats a bigger multilingual one.

• The lean: PolDense-68M (OPI-PIB) — 68M-param ModernBERT that matches BGE-M3 (568M) and Jina-V5 on the PIRB benchmark (41 Polish retrieval tasks) at ~1/10th the size, ~300 MB, CPU-fast (the 3090 stays committed to llama-swap). Sibling PolDense-400M matches BGE-Multilingual-Gemma2 and Llama-Embed-Nemotron-8B — the escalation path if retrieval quality bottlenecks.
• Two-model strategy: specialized-small for Polish-first content + BGE-M3 (568M, MIT, 8192 ctx, Qdrant’s default rec) as the multilingual fallback. Candidates evaluated and set aside: Snowflake-Arctic-Embed-2.0 (568M MoE), Nomic-Embed-v2 (305M MoE, weaker Polish), BGE-small-en (rejected — EN-only), Qwen3-Embedding 4B/8B (top MTEB, overkill for homelab RAG).
• The license gotcha worth remembering: PolDense is Gemma-licensed, not OSI — the distillation teacher’s license (BGE-Multilingual-Gemma2) carries to the student. Fine for personal use; a blocker the day the embedding service goes public.
• Open questions still standing there: ONNX export for PolDense (Hindsight’s CPU path), TEI vs Qdrant-native embeddings, dual per-doc embeddings (PL+EN) vs one multilingual model.

2026-10 update from this note’s capture: EmbeddingGemma 2 (multimodal, 0.5 GB) joins that candidate table as the unified-space option — but it doesn’t answer the Polish question; PIRB-grade PL quality is still PolDense’s moat. The homelab decision tree stands: lexical baseline → PolDense-68M PL lane → BGE-M3/EmbeddingGemma 2 multilingual+multimodal lane.

The 77× squeeze: TurboQuant + MRL on EmbeddingGemma 2 (Qdrant, 2026-10-06)

KShivendu_ (628❤ / 25k views; 2-tweet + Qdrant’s full benchmark post, both captured): 10M EmbeddingGemma 2 vectors from ~30 GB (float32, 768d) → ~1 GB (1-bit TurboQuant, 99% nDCG@10) → ~0.4 GB (+ MRL truncation to 256d, 94.5% with rescoring) — 77× less RAM for ~5% quality.

The recipe, in Qdrant’s recommended order:

1. Boring-first: full 768 dims, 1-bit TurboQuant, rescoring off — 104 bytes/vector vs 3,072; <half the float32 query time, 99.0% quality retained. (74% of top-10 match exact search exactly; add rescoring only if identical neighbors matter.)
2. Then truncate: 256 dims + 1-bit + rescoring = 40 bytes/vector, 94.5%.
3. The counterintuitive bit: cut bits before dimensions — 512d@1-bit (72 B/vector) kept 97.3% vs 128d@4-bit at a similar budget keeping only 84.0%. Compress each dimension harder first; shorten vectors last.

Homelab relevance (the flagged experiment): the homelab roadmap’s RAG lane sizes embedding RAM at 0.3-1.5 GB — this makes even the multilingual+multimodal model practical next to PolDense: same trick applies to any model (MRL dims + TurboQuant bits are storage-side, model-independent). Kin: the Nix - package manager-grade lesson — quantize the index, not just the model; Matryoshka note above (design axes) now has field numbers.

pplx-embed-v2-late: late-interaction arrives for documents (2026-10-08)

lmoroney (114❤; Laurence Moroney, ex-Google AI devrel): Perplexity open-sourced pplx-embed-v2-late (MIT, HF; 0.6B and 9B sizes on Qwen3.5) — ColBERT-style late-interaction retrievers: every token gets its own 128-d vector, queries score by MaxSim against per-token document vectors. The payload: “built to skip OCR” — they ingest text, images, and rendered pages (PDFs, slides) directly, unifying the modalities in one index. Both sizes share one embedding space, so 0.6B queries can hit a 9B-built index (the asymmetric index/query split — same idea as the Qdrant MRL recipe in this note). ViDoRe v3 image track: 62.3% nDCG@10 (0.6B), 65.2% (9B); both distilled from an internal 18B teacher. The honest caveat from the tweet: storing a vector per token makes indexes big — test on your own data first (few hundred pages with the 9B, query with both, measure disk). For this note’s design axes: late interaction is the third paradigm alongside single-vector dense and sparse — it keeps token-level granularity (fixing the “lost in the middle of one vector” problem) at the cost of index size; and document-as-rendered-page input is the multimodal-unified-space lane going past text+image into layout. Homelab lens: index-size-vs-quality tradeoff belongs in the candidates ledger next to EmbeddingGemma 2 + TurboQuant.

Embedding models as decision heads: Typed Decision (2026-10-12)

Manu Romero (@mrm8488) (298❤/13.7k views; tweet + model card captured): EmbeddingGemma 2 fine-tuned into a typed decision model — give it a state (agent trace, support ticket, invoice, security alert) + a schema of questions, and it returns typed, calibrated answers: choice (str), score (int 0..3), noul (probabilistic bool), each with confidence and full probability distribution. 76.4% on typed-decisions, ~6 pts behind a 27B LLM with ~100× fewer params. No text generation, no output parsing.

The trick that generalizes: the head scores the text of each option against the state and the question — so the question schema is plain data. Rename options, change criteria descriptions, add new questions without retraining (far-from-training-data questions are less reliable — the card is honest about it). Pipeline tag: text-classification, not generation.

Why this belongs in the embedding lane: it’s the judge-model idea (What matters in AI world’s “judge over worker”) taken to its endpoint — decisions as embeddings + option scoring instead of generation + parsing. Same family as the LLM-evals “judge is a classifier” validation doctrine, and the LocalAI “model router with decision models” capture: small trained deciders routing/gating the big generators. For the homelab ledger: a ~0.3B Apache-2.0 decision head over llama-swap generators is a realistic local gate for trace triage (continue/observe/human_review/stop) — the exact schema in the card’s example.