AI engineering

The engineering discipline around LLMs and agents: not training frontier models, but building reliable systems on top of them. The craft splits into planes that conventional software never had to worry about:

• Context plane — token budgeting between memory, tools, and docs per request; context as a scarce, groomed resource rather than ambient memory. Kin: Agent knowledge systems.

• Control plane — crash-resilient workflows (state machines over fragile chat loops — the State machines/durable-execution convergence), kill-switches, degradation chains, human-in-the-loop gates for high-stakes actions.

• Trust plane — evals before vibes (LLM evals, Agent tests trajectory grading), prompt-injection containment between agents, PII masking proxies, sandboxing (Agent sandbox).

• Efficiency plane — routing fast/simple queries to cheap models and escalating edge cases to reasoning models, KV-cache prefix sharing, speculative decoding, semantic caching; see Local LLM inference for the local end of that ladder.

• Feedback plane — shadow traffic canaries, auto-generated trajectory diffs, thumbs-down → preference pairs → nightly LoRA fine-tune (LLM fine-tuning); model quality as a pipeline, not a release.

• Trust plane, addendum — small guardrail models: Qwen3Guard (OSS) (Qwen team, 515★; via @mfranz_on: “small, real-time safety layer alongside your LLM… this and a sandbox = a real security environment”). 0.6B/4B/8B series trained on 1.19M labeled prompt/response pairs; two variants — Gen (prompt+response safety classification) and Stream (token-level classification head for real-time moderation during generation, no post-hoc pass); safe/controversial/unsafe three-tier severity; 119 languages; SOTA on streaming-moderation benchmarks. The 0.6B Stream variant is the local-inference sweet spot (runs beside any LLM on modest hardware — llama-swap-class setups; even MLX mxfp4 community builds). The pattern: containment stops at the sandbox boundary; a guard model covers what the sandbox can’t see — generated content risk (the LLM evals lane’s runtime sibling).
  The field’s tell: most of the work is measurement, containment, and fallbacks — the model is the easy part you wrap in engineering. Musk engineering process applies unchanged (delete prompt-path complexity before optimizing it), and Operating principles (“engineer systems with safety controls”) is arguably the AI-engineering job description.

Concrete practice targets, sourced and annotated: AI engineering stretch goals.

Related: Vibecoding vs AI engineering (the discipline boundary) · Agent Client Protocol · Hermes Agent (a working instance of these planes)

The enterprise reality check (Varick Agents’ field report)

Varick Agents (SF consultancy deploying agent systems inside Fortune-100 companies; blog surfaced via @ace_the_agent, captured: the index + “If AI is so great, why isn’t it working?” by Daniel Kornum — 100+ C-suite conversations). The diagnosis: models are good enough, but away from engineering every vendor-promised category shows flat metrics (AP unchanged, close still 22 days, quota still 24%, same CRM decay). Why engineering works and nothing else does — four properties no other enterprise function has: bounded, checkable, structured substrate, verifiable output. The four failure modes they see every time: skip-the-audit · throw-everything-at-LLMs · agent sprawl · AI-as-side-project-not-infrastructure. The 5% that ships and stays: (1) audit before build — 4+ weeks mapping the real workflow into a digital twin (alignment > document); (2) decompose until most steps are deterministic — “LLM goes ONLY where judgment is required; most production systems end up as 5-10 deterministic steps with one or two model calls. Boring in production is the goal”; (3) single orchestration layer over the existing stack (“single pane of glass”, sprawl dead on arrival); (4) model-agnostic routing — abstractions at task level, routing absorbs model churn. Plus their FDE 101 guide (Eyad Khrais): “if intelligence is becoming commoditized, there is no competitive advantage in intelligence alone — only in how and where you use it,” which is the forward-deployed-engineer role thesis. Vault fit: the 5-10-deterministic-steps pattern IS the do-nothing/KISS arc at enterprise scale, and property #2 (checkable) is exactly where Decision models and the verification cluster insert checkability into non-engineering work.