AI metrics that actually matter
The unit of work is the task, not the token. Metrics that divide by tokens measure the engine; the user experiences the journey.
The two user-facing metrics
Time per task — wall-clock from request to acceptable result. The only latency the user feels. Dominated by things tokens-per-second ignores: reasoning length (tokens × 1/speed), tool loops, TTFT, overheads, retries.
Cost per task — dollars (or metered compute) per acceptable result. Not $/1M tokens; the tokens are the means, not the work. Same trap as time: a cheaper per-token model can cost more per task by being verbose, or by needing retries/escalation.
Everything else is an engine metric — real, but intermediate.
The anchor capture: tokens-per-second is meaningless
Adam Holter (@AdamHoltererer), 2026-10-05 — 1,291❤ / 146k views: “Hot take: Tokens per second is a completely meaningless metric” + 20s animation (frames vision-transcribed into the clipping). The race: Opus 5.5 at 71.5 tok/s vs Gemini 3.8 Flash at 239.1 tok/s (3.34× faster decode). Result: Opus FINISHED at 88.12s; Gemini still DRIVING at 146.72s → 294.05s — the fast car on the long road loses to the slow car on the short road. Footnote honesty: “AA decode averages. Excludes TTFT and overhead” — granting the fast model every decode advantage, it still loses, because speed × distance = outcome and the distance is the model’s own token count. Users experience time to completion, not throughput.
Why engine metrics mislead (the four traps)
- Distance isn’t fixed — the model chooses its own token budget; verbosity multiplies the bill in both time and cost. (The Holter race is exactly this trap.)
- The loop is the unit — agent tasks are tool-loop latencies: TTFT + decode + tool exec + retry, repeated. Decode is one factor of one term. (Inference engines: prefill/decode split; speculative decoding exists precisely because decode dominates wall-clock.)
- Benchmark-vs-production drift — sandbox metrics (tok/s, % solved) measure the engine on synthetic distance; production adds retries, escalations, verification. The famous “100% in eval, 60% live” gap.
- Cost of supervision is part of cost per task — review/verification time is user time; Spec Gate’s economics (green CI ≠ cheap run) live here.
Vault kin
- Vibecoding vs AI engineering — the retro’s quantitative eye; efficiency as a first-class eng concern, not vibes.
- Decision models — where quality gates and token counts trade off.
- Small language models — “one job at frontier-adjacent quality” = per-task cost optimization by model routing.
- CursorBench Pareto (illyism chart) — % solved vs $/task is this plane; Sonnet 5.5 High = 85% of Opus for 12% price.
- answer-me-with-html — token-economics design (6.1× fewer output tokens = direct time/cost-per-task win); the 3b1b mode pushes the same axis.
- Leviathan — cited result cards (~450 tokens/answer at any row count) as retrieval-side token discipline.
- nykdotdev Spec Gate — “green CI does not mean the agent run is cheap”: review-side metrics beyond tokens.
- Inference engines — KServe/jubril fleet view: autoscaling & cost operate at serving time, per task.
- Jim O’Shaughnessy / luck-surface framing — more shots on goal; per-shot cost defines shot volume.
Practice
- Measure tasks/hr and $/task for recurring workflows (oncall bundles, garden captures) — not tok/s.
- Logs that already exist contain both: session durations + provider usage rows.
- When a benchmark shows speed or $/1M-tokens, ask: at equal task quality? At equal retry count? Distance equal?
- For the homelab: llama-swap slot choice is a cost-per-task decision (GLM-4.6-Qwen swap latency vs quality tier), not a throughput one.