AI engineering stretch goals

A ranked backlog of build-it-to-learn-it projects for the AI engineering planes — concrete practice targets rather than reading material. Seed source: @suraj_sharma14 “CANCEL your weekend plans” (verbatim clipping: clippings/x-2105991192349679919.md, 19 items) — the engagement-bait framing stripped, the ideas kept.

Control plane

• Temporal/state-machine agent workflows (crash-resilient) → State machines, Pi Durable kin
• Cost kill-switch: halt any agent loop when projected spend exceeds a threshold
• 4-tier graceful degradation: Frontier API → mid-tier → local quantized → semantic cache
• Human-in-the-loop approval gates (Slack/Telegram) for high-stakes actions — Hermes tool-approval is a live instance

Trust plane

• Inter-agent sanitization proxy (Agent A output can’t prompt-inject Agent B)
• PII unmasking proxy: swap sensitive data for UUIDs pre-LLM, restore post-generation
• Trajectory grading in CI: block PRs when tool-call sequences deviate from the golden path → Agent tests
• Fuzz the agent swarm: malformed tool outputs, context overflow, self-healing recovery tests

Efficiency plane

• Model router: fast “System-One” model handles ~95% triage, escalates edge cases to reasoning models — local instance: llama.cpp decision models (/v1/systemone endpoint, 2026-10-02): question+choices in, probabilities out, no generated answer. Julia-1 at 144M answers in ~3 ms — the router as a plain GGUF on the same llama-swap box, no separate service. Captures: clippings/x-2106074231494852863.md (TeksEdge announcement) · clippings/huggingface-co-blog-ggml-org-decision-models-in-llamacpp.md (ggml-org blog: model table, latencies, quick start llama serve -hf ggml-org/Kev-4B-GGUF) · the Decision Index reality check (clippings/multimodalart-jev-decision-index-static-hf-space-index-html.md): 70 models × 54 benchmarks, community board. Julia-1’s balanced_skill ≈ 5.5 vs ≈ 57 for the 26–32B leaders (Rune 26B-A4B, Decider chat 31B) — the 3 ms router is a cheap triage gate, not a quality decider; routing to the big model stays the default for anything consequential. Index payload archived raw at _research/references/decision-index-2026-09-28.json
• KV-cache prefix sharing at the proxy layer (TTFT cuts for identical system prompts) — local benchmark case: buun-llama --resume (Inference engines): persists warm KV, slots, checkpoints, drafter states and the host prompt cache to disk on exit, reloads on start — “kill cold prefill”. --resume-no-host-cache gives the slots-only variant for A/B isolation. Benchmark protocol: fixed long prefix (big system prompt / long doc), measure TTFT cold-start vs --resume restart vs --resume-no-host-cache; that quantifies exactly what a proxy-layer prefix cache would buy, on the doombox 3090 through llama-swap today (the new --slot-save-path format is llama-swap-automatic). Distinction to keep straight: --resume = persistence across restarts; vLLM PagedAttention/SGLang RadixAttention = sharing across concurrent requests — related mechanisms, different experiments
• Speculative decoding: local 2B drafts, cloud 70B verifies (60% latency cut claim)
• API→browser fallback: headless browser spawn when REST 404s (Hermes browser stack as reference)

Context & memory plane

• Dynamic context assembler: strict token budget across memory/tools/docs per request
• 3-tier memory engine (working/episodic/semantic) with automated eviction + compression → Agent knowledge systems

Feedback & integration plane

• Shadow canary: route 5% traffic to a new model, auto-diff trajectories
• DPO pipeline: thumbs-down → preference pairs → nightly LoRA fine-tune → LLM fine-tuning
• Raw MCP server from scratch exposing a database securely → Agent Client Protocol kin

KRZ-relevance triage (homelab-scale)

Highest learning-per-effort against the existing stack: model router, cost kill-switch, trajectory grading in CI, MCP-from-scratch. The KV-cache/speculative-decoding items need proxy-layer ownership most homelab setups lack; the DPO pipeline is the largest single build.

Related: AI engineering · Agent tests · LLM evals