Local LM roadmap

Imported 2026-09-13 from the private homelab repo (docs/local-lm-roadmap.md,
§2 + §2.1 + §3) — byte-identical copy. Context: Homelab, QMD (which
obviates most of §2’s open questions for the garden vault).

2. RAG pipeline → local Markdown knowledge base

Goal: Ingest documents (PDFs, web pages, notes) into a local Markdown-based
knowledge base, queryable via embeddings + Obsidian/obsidian-cli.

Architecture idea:

Document sources        Processing           Storage              Query
┌─────────────┐       ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│ PDFs        │──────▶│ Chunking     │───▶│ ~/vault/     │───▶│ Embeddings   │
│ Web pages   │       │ + summarize  │    │  kb/         │    │ + vector DB  │
│ Notes dump  │       │ + metadata   │    │  *.md files  │    │ → obsidian-cli│
└─────────────┘       └──────────────┘    └──────────────┘    └──────────────┘
                                                 │                    │
                                                 ▼                    ▼
                                           Obsidian UI          LLM-grounded
                                           (manual browse)      retrieval

Components:

• Storage: Markdown files in an Obsidian vault (~/vault/ or existing).
  One doc = one .md file with YAML frontmatter (source URL, date, tags).
• Querying (basic): Obsidian CLI / obsidian-cli skill for direct file
  search, grep, wikilink traversal.
• Querying (semantic): embedding model + vector DB for similarity search
  over chunked content. Chunks reference back to source .md files.
• Ingestion pipeline: document → text extraction → chunking → summarization
  (via llama-swap) → write .md → embed chunks → store vectors.

Why Markdown + Obsidian:

• Human-readable, portable, version-controllable ( Forgejo).
• Obsidian provides manual browsing + graph view on top.
• No vendor lock-in — files outlast any tool.

Open questions:

• Which embedding model? See Embedding model comparison below.
• Vector DB: Qdrant (container), LanceDB (embedded, no server), or pgvector
  (if PostgreSQL lands for Hindsight)?
• Chunking strategy: fixed-size, sentence-aware, or document-structure-aware?
• Obsidian vault location and structure (new vault vs existing)?

Embedding model comparison

The choice of embedding model affects RAG (§2), memory systems (§3 —
Hindsight/Supermemory/agentmemory all use embeddings), and any semantic-search
workflow. Most homelab content will be mixed Polish/English, so a
multilingual model (or a Polish-specialized one) is needed — pure English
models will fail on Polish queries.

Candidate models

Model	Params	Langs	Polish quality	License	VRAM/RAM	Notes
PolDense-68M (OPI-PIB)	68M	PL (specialized)	🟢 Best-in-class for size¹	Gemma²	~300 MB	ModernBERT. Matches BGE-M3/Jina-V5 quality at 1/10th the size. Lean for PL-heavy RAG.
PolDense-400M (OPI-PIB)	400M	PL (specialized)	🟢 SOTA for PL¹	Gemma²	~1.6 GB	ModernBERT. Matches BGE-Multilingual-Gemma2. For max PL retrieval quality.
BGE-M3 (BAAI)	568M	Multi (100+)	✅ Good	MIT	~2 GB	Qdrant’s default recommendation. Multilingual, long-context (8192).
Snowflake-Arctic-Embed-2.0	568M	Multi	✅ Good	Apache 2.0	~2 GB	Qdrant-recommended multilingual. Mixture-of-Experts.
Nomic-Embed-v2	305M	Multi	✅ OK	Apache 2.0	~1.2 GB	MoE. First MoE embedding model. Good English, weaker Polish.
BGE-small-en	33M	EN only	❌ None	MIT	~130 MB	Smallest. English-only — rejected for PL content.
Qwen3-Embedding (4B/8B)	4B/8B	Multi	✅ Very good	Apache 2.0	8–16 GB	Top MTEB scores but heavy. Overkill for homelab RAG.

¹ PolDense models were evaluated on the PIRB benchmark
(41 Polish retrieval tasks). Per OPI-PIB: the 68M variant matches multilingual
models several times its size (Jina-V5, BGE-M3, Snowflake-Arctic-2.0); the 400M
matches BGE-Multilingual-Gemma2 and Llama-Embed-Nemotron-8B.

² PolDense license: Gemma. Not OSI-approved — the distillation teacher
(BGE-Multilingual-Gemma2) is Gemma-licensed, which carries to the student.
Fine for personal/homelab use; a consideration if we ever expose the embedding
service publicly.

Lean: PolDense-68M for Polish-first content, BGE-M3 as multilingual fallback

Two-model strategy:

1. PolDense-68M as the default for RAG (§2) and memory systems (§3) — the
   homelab’s knowledge base will be primarily Polish. 68M is small enough to
   run on CPU (no VRAM contention with llama-swap), and quality matches models
   10× its size on Polish retrieval (PIRB benchmark).
2. BGE-M3 as a multilingual fallback if we ingest significant English/other
   content where a specialized Polish model underperforms. Can run on GPU
   (~2 GB) or CPU.

Why PolDense-68M over the larger 400M:

• CPU-friendly — 68M params runs fast on CPU, no VRAM needed. Critical
  because the RTX 3090 is committed to LLM inference.
• Quality parity with much larger models on Polish — the PIRB results show
  68M matches BGE-M3 (568M) on Polish retrieval.
• ModernBERT architecture — efficient, well-supported by sentence-transformers.

When to consider PolDense-400M:

• If RAG retrieval quality on Polish becomes a bottleneck (68M insufficient)
• If we have VRAM headroom (run on GPU alongside or in place of llama-swap)
• For the “best possible” memory system retrieval

Memory systems and their embedding defaults

Different memory systems (§3 comparison) ship with different default embedding
models — need to verify whether each can use PolDense or requires its own:

Memory system	Default embedder	Can swap to PolDense?	Notes
Hindsight	Configurable (ONNX or PyTorch)	✅ Likely (sentence-transformers compatible)	Verify ONNX export path for PolDense
Supermemory	?	?	Research needed (part of §3 comparison)
agentmemory	?	?	Research needed

Resource estimate

Model	RAM (CPU)	VRAM (GPU)	Latency
PolDense-68M	~300 MB	~300 MB	Fast on CPU (ModernBERT efficient)
PolDense-400M	~1.6 GB	~1.6 GB	Medium on CPU
BGE-M3	~2 GB	~2 GB	Fast on GPU, medium on CPU

Open questions

• Does PolDense export cleanly to ONNX (for Hindsight’s ONNX path / faster CPU)?
• Can we run PolDense-68M on CPU alongside the existing CPU services
  (voiceapi, chatterbox) without RAM pressure?
• Does Qdrant have a built-in PolDense integration, or do we run a separate
  embedding server (text-embeddings-inference, TEI)?
• Mixed-language content: do we need dual embeddings (PL + EN model per doc)
  or does a single multilingual model (BGE-M3) suffice?

Status: PolDense-68M is the lean for Polish-heavy RAG. Needs ONNX export
verification + integration test with Hindsight/Qdrant.

Spec: TBD

3. Memory system comparison

Goal: Choose a memory system for Hermes Agent (and potentially other agents)
that provides structured fact extraction, semantic retrieval, and consolidation.

Needs research — three candidates:

System	Type	Self-hosted?	Notes
Hindsight (Vectorize)	Biomimetic memory	✅ Docker (slim image)	SOTA on LongMemEval. Native Hermes integration. Spec 0003 drafted.
Supermemory	Memory layer for AI apps	✅ Docker	?
agentmemory	Agent-focused memory	?	?

Required deliverable: _research/memory-systems-comparison.md —
in-depth comparison covering:

• Architecture (embedding model, retrieval method, storage backend)
• Resource footprint (RAM, VRAM, disk, CPU at idle and under load)
• API surface (REST? gRPC? OpenAI-compatible?)
• Hermes Agent integration maturity
• Self-hosting complexity (Docker? bare metal? PostgreSQL dependency?)
• Scalability and data portability
• Polish/multilingual support (for non-English content)
• License and project maturity

Research complete: _research/memory-systems-comparison.md —
in-depth comparison of Hindsight, Supermemory, agentmemory (deep dive) + Mem0,
Letta (summarized). Covers architecture, resource footprint, API, Hermes
integration, multilingual/Polish support, benchmarks, and a homelab-fit
decision matrix.

Three primary candidates:

System	Self-host	Open source?	Hermes	Polish	Ops
Hindsight	Docker + Postgres	✅ MIT	✅ Built-in	✅ Excellent	Medium
Supermemory	Single binary	❌ Engine closed-source	✅ Built-in	⚠️ Swappable	✅ Simplest
agentmemory	npm / iii-engine	✅ Apache-2.0	✅ 6-hook plugin	⚠️ English-default	✅ Low

Status: Research complete. Next: pick 2 finalists for hands-on testing with
a real Polish/English corpus, then deployment decision.

---