LLM evals

Evals are the testing discipline for probabilistic software: structured tests that score what a model/agent produces against a defined expectation — a reference answer, a policy rule, a functional check, a rubric. The core doctrine (Hamel/Shreya school, captured FAQ): error analysis first — read failing traces by hand before building any automated evaluator, and a score is only useful when it changes what ships (becomes a dataset example, an online monitor, or a context fix).

Benchmarks ≠ evals

Benchmarks (MMLU, SWE-bench, GPQA…) measure a model’s raw capability on a shared fixed set — right tool exactly once, when choosing a base model. Evals score your whole system (prompts, tools, retrieval, policies, your data) on the behavior you care about, continuously as the system changes. Conflating the two is the category error behind “it benchmarks well, it’ll be fine in prod”.

Why agents break normal testing

• Same input → different acceptable outputs; exact-match checking flags good replies as wrong
• No single correct answer — judge against a description of good, not one reference
• New failure shapes: confident falsehoods, instruction manipulation, policy violations, wrong tool order — none crash, all pass functional tests

So: grade against written checklists or LLM-as-a-judge, not exact match. Same conclusion as Agent tests from the code side.

The lifecycle position (ADLC)

Evals work at two points of the agent development lifecycle (LangChain’s framing, captured):

• Test (pre-deploy) — curated datasets, expected behavior, version-vs-baseline regression gating
• Monitor (post-deploy) — production traces, reference-free judging, drift + recurring failure modes; failing traces get promoted into datasets so a prod issue becomes a testable example, not a screenshot in a ticket

Test data is written by you; monitor data is whatever users actually did. Different setups, different evaluators — offline (reference-based judges, code checks, human review) vs online (reference-free judges, policy checks).

Name failures before scoring them

Vague metrics (“helpfulness”, “hallucination”) hide disagreement. Trace review ends with three outputs: a named failure mode, labeled examples, the evaluator that can observe it. Named failures look like “rescheduled a tour when no slot existed”, “called the refund tool before verifying eligibility” — actionable, evidence-based. Expect the rubric to drift early; that’s the team learning what it should have said.

Evaluator matching: code/functional (deterministic failures, cheap, every run) · LLM-as-a-judge (semantic failures, needs calibration against labeled traces) · human review (calibrates the judge, golden datasets, high-stakes domains). Same trio as Agent tests’ toolchain.

The judge is a classifier

The unifying insight (Hamel’s FAQ): an LLM judge is just a classifier — validate it like one, against trusted human labels, with train/dev/test separation, before trusting its scores to gate anything. This dissolves the mystery: Decision models (Jev-class fast classifiers) and Fine-tuning GLiNER (small fine-tuned classifiers for known labels) are both “judge” candidates, validated identically. Binary pass/fail beats 1-5 Likert (easier calibration); similarity metrics (BERTScore/ROUGE) are mostly theater; don’t use the same model for task and judge without checking agreement; cheap judges (Jev, GLiNER) make automated prompt tuning (GEPA-style, hundreds of evals) affordable.

Practice pointers

• Minimum viable setup: a handful of examples + human review loop, before any tooling
• Eval-driven development: write the eval alongside the feature, like TDD for prompts
• Don’t automate every failure mode — error analysis tells you which ones matter
• Promotion loop: prod failure → labeled example → regression coverage → online monitor (LangSmith Engine productizes this as the issue-attached loop: cluster traces → diagnose → fix → suggest evaluator + dataset examples)

Related: Agent tests (the code-test sibling; tautological-test warning applies to evals too — an eval that asserts the model’s own output proves nothing) · Decision models · Fine-tuning GLiNER · Property-based testing (invariant-generation mindset for the “no single right answer” problem)

The curriculum form: sermakarevich’s plain-language evals article (2026-10-06)

sermakarevich + the full X article quoted whole in both captures (53.6 KB — “Evals: how to know whether an AI system actually works”; “for everyone who ships, buys, or signs off on software built on LLMs — engineers, PMs, and the CEO”). The framing this note lacked: “An LLM almost never fails loudly… If it reads well, it must be working” — vs data science’s opposite assumption: nothing works until you prove it. And the sharpest field observation of the skill-madness era: millions of prompts/skills/instruction files on portals, “almost none come with any evidence they help… tuned against a model that is already outdated, and never run against a single test case. People install them, the output still looks magical, and the cycle continues.” The fix stated in one line: build a test set, measure, apply the change, measure again — if the number doesn’t move beyond run-to-run variation, it didn’t help.

The article’s ladder (zoom one level at a time, each answering one question about the level above):

• L0 the whole loop on one ticket — two cheap code checks passed, the model-judge caught an invented “15 business days” allowance: the grader is part of the system you must test
• L1 what we measure — inputs + gold answers; one number per experiment, always on the test set, always with an error bar
• L2 look at the data first (error analysis — the Hamel doctrine, agreed)
• L3 graders made of code — cheap, exact, blind to meaning (17% of replies fit in 150 words — no judge needed; IFEval = all verifiable rules)
• L4 model-as-grader — flexible, biased, must be checked (“the most common way an AI team fools itself is a confident number from an unchecked grader”; two numbers to ask for: how good the product is, how good the grader is)
• L5 honest numbers — error bars, A/B, how many cases
• L6 systems with parts — retrieval, agents, cost, reliability
• L7 public benchmarks — what leaderboards do and don’t tell you
• L8 making it a habit — gates, monitoring, when to re-check

Every number sourced (runnable bike-shop-support tutorial or cited paper): LettuceDetect, MiniCheck, SelfCheckGPT, HELM, G-Eval, Adding Error Bars to Evals (Miller), ARES, Emergent Abilities a Mirage?, Who Validates the Validators?, JudgeBench, PoLL (juries of judges over single judges), LLMs are not Fair Evaluators, Eugene Yan’s task-specific evals, τ-bench/τ²-bench, GSM1k (contamination), MT-Bench/Chatbot Arena, IFEval, + 2026 audits (benchmark plateau, judge drift, reliability-without-validity, AgentLens).

The everything-open-source demo: observability + evals as one loop (ShenSeanChen)

ShenSeanChen (2-tweet + 19-min video w/ chapter index in the clipping): the framing is the keeper — “observability is a camera; eval is a referee. The camera records what happened; the referee decides whether the run was good enough to ship.” Ran end-to-end on two open-source repos (both captured): treg (OSS) (4.3k★ Python — “OpenRouter for agent tools”: 3,800 tools behind one API key, every call goes through a meter = the camera) and waku-agent (OSS) (1.9k★ Python — local-first harness “you actually own, including loop, memory, eval, all in code built to stay legible”; portable memory callable from Claude Code = the referee’s state). The demo’s numbers: ten tools, 29 cents, zero errors; six evals on one run; release gate = 3,218 deterministic evals; a second run includes a failed tool call graded on camera. Kin: the camera/referee split is this note’s online/offline pair stated physically; treg’s one-key-many-tools is the CLIs for agents inventory problem at wire level; the gateway-meter lane connects to Agent-accessible product.