Agent tests

Current models write a lot of unit tests, and most of them are tautological: written after the code, they restate the implementation, so they always pass, catch almost nothing, run fast, and break on every refactor — after which the agent burns its time fixing tests instead of the feature. The failure mode is structural, not a prompting slip: a test derived from the code it shadows can only ever confirm the code back to itself.

The counter-pattern that works, from @RayFernando1337 (2932 ❤, quoting @anshnanda), is three AGENTS.md rules for coding agents:

• Never write unit tests after you write code. Test-first or nothing — a test written after the code is drawn toward restating it.
• Highly prefer E2E tests as the sole testing mechanism. Use them to verify complex features work; at the end, produce a verifiable, repeatable artifact.
• If you must test a system in isolation, first write down all the ways it could fail, then write the code. The failure list is the test design — not the implementation’s shape.

The cleanup prompt that goes with it: delete every low-signal unit test that wouldn’t catch a real bug the E2E tests miss, fanned out across parallel subagents, then pin the rules in AGENTS.md so future agents stop regenerating the debt.

This is the same principle as the classic “no tautological tests” rule — expected values come from a spec or hand-computed result, never from the code’s own output — applied to agents, where the failure rate is higher because generating plausible-looking tests is exactly what LLMs are good at.

Source: tweet captured 2026-09-25, clippings/x-2102927565778522610.md (includes the quoted @anshnanda original).

Related capture: @mattpocockuk — “No tautological tests”, quoting @dexhorthy’s Opus adding 10 unit tests to assert substrings of a constant string (clippings/x-2102088071244054697.md).

Growing

• Where is the line for infra/CLI repos where E2E is slow — failure-mode-first isolation tests as the compromise?
• Does verifiable-artifact-at-the-end generalize to Hermes Agent harness work (tool output as the artifact)?
• The model-behavior sibling of this note is LLM evals — same doctrine from the other side: error analysis first, named failure modes, judge-is-a-classifier validation; the tautological-test warning applies to evals unchanged.

swizec’s version (97%-AI codebase): “In the words of Knuth, ‘Beware of bugs in the above code; I have only proved it correct, not tried it.’ I don’t trust code I didn’t see working with my own eyes” — even with agents generating screenshots and walkthrough videos, a happy-path smoke test before merge is non-negotiable; it doubles as the cheapest UX-friction check before shipping.