Write the failing fixture first or you have written a description

A gate is written to catch a defect. The defect is fixed in the same change. The gate passes, and it goes in.

Nobody has established that the gate would have caught anything.

The symptom#

A check that has only ever been green. It passes on the current code, which is the code that no longer has the problem, and that observation is consistent with two very different situations: the check works and the defect is gone, or the check does not work and never would have fired.

The second thing is that the two situations are indistinguishable from the outside, permanently. A passing check produces the same output whether it is vigilant or inert, and a check that has never fired accumulates confidence over time rather than suspicion — a gate that has been green for a year reads as a solved problem rather than as an untested instrument.

The third is how easy it is to write an inert one. A path that does not match. A pattern that silently finds nothing. A scan over a directory that was renamed. A condition whose branch is never reached. None of these produce an error, because finding nothing is a perfectly ordinary outcome for a check that is looking for a problem.

The fourth is the specific confidence this produces. People stop looking for the defect class, because something is watching for it. The gate has not merely failed to help; it has displaced the attention that would otherwise have been there.

That displacement is the part that turns an inert check from useless into actively harmful. Before the gate existed, the defect class was something people occasionally looked for; afterwards it is something that has been handled, and the manual vigilance that was imperfect but real is gone.

Why the obvious fix failed#

Review the gate carefully. Reads the logic and cannot establish that the logic matches reality. Most inert gates are logically correct and factually wrong — a path that is right in principle and does not exist, a pattern that would match the thing if the thing were spelled the way the author remembered.

Test the gate's code. Useful for anything with real logic and usually misses the failure. The bug is in the interface with the world — what it scans, where it looks — and a unit test of the comparison function does not touch that.

Run it against the whole codebase and see if it finds anything. Promising, and silent when the codebase is clean. Finding nothing is exactly what a working gate does on healthy code, so the experiment has no negative result.

Trust that it worked when it was written. Sometimes true, and it does not survive the first time a path is reorganised. Inertness is also acquired: a gate that worked for a year and then had its target directory renamed is now exactly the inert case, with a year of green runs behind it.

The actual mechanism#

Build the failing case first, watch it fail, then fix it and watch the same check pass.

red     a fixture that genuinely contains the defect — gate must fail on it
fix     the change that removes the defect
green   the same gate, same fixture family, now passing
keep    both fixtures, so the gate is re-proven on every run

The fourth line is what makes this more than a development ritual. A red fixture kept in the repository is a permanent, mechanical answer to "does this gate still detect anything" — and it converts gate-inertness from something nobody can observe into something that fails loudly the moment it happens.

The ordering is not a style preference. Writing the fixture after the fix means writing it against a clean tree, where there is nothing to fail on, and the author has to imagine the defect rather than reproduce it. Imagined defects are usually imagined in the same shape as the author's mental model, which is also the shape the gate was written against — so they agree with each other and neither agrees with reality.

This is also why "I'll add the fixture afterwards" is not the same thing deferred. The information the red run carries only exists while the defect does; once the tree is clean, reproducing it means reconstructing it from memory, and the reconstruction inherits every assumption that produced the gate.

The sub-case that is worth getting right is one fixture, one class. A red fixture containing three different violations fails if any of them is detected, so a gate that catches one and misses two reports red and looks vigilant. The fixture has established that the gate detects something, which is not what anybody wanted to know.

The same logic applies to a fixture that is red for an unrelated reason — a syntax error, a missing file, a path that does not resolve. The gate fails, the run is red, and nothing has been established about detection at all. A red fixture needs to be red for exactly one reason and that reason has to be the defect.

The fix#

isolate   one red fixture per defect class, containing exactly that violation
prove     run the gate on it before the fix exists; it must fail
pair      a matching green fixture that is identical except for the defect
rerun     both fixtures execute in CI, so the proof is continuous

Two directories and a nine-line script. The fixtures are a red/ tree containing exactly one violation and a green/ tree identical but for that violation:

tests/fixtures/fork-module-seam-canary/
  red/     settings.gradle.kts          <- the fork include in the TEMPLATE-OWNED file
  green/   settings.gradle.kts          <- the same include, moved to the seam
           settings.local.gradle.kts

#!/usr/bin/env bash
# tests/run-canary.sh <gate-script> <canary-dir>
#
# Runs the gate twice and compares exit codes. That is the whole harness — the discipline is
# the ordering and the isolation, not the tooling.
set -uo pipefail                      # NOT -e: a non-zero exit is the expected result below
GATE="$1"; DIR="$2"

bash "$GATE" --root "$DIR/red"   >/dev/null 2>&1; red=$?
bash "$GATE" --root "$DIR/green" >/dev/null 2>&1; green=$?

# RED must fail. A gate that passes its red fixture is inert, and this is the only run that
# can tell you so — on a clean tree it would pass for the same reason a working gate does.
if [ "$red" -eq 0 ]; then
  echo "INERT: $GATE passed $DIR/red — it detects nothing"
  exit 1
fi

# GREEN must pass. Without this half, a gate that fails on everything also satisfies the check
# above, and "always red" is as uninformative as "always green".
if [ "$green" -ne 0 ]; then
  echo "OVER-BROAD: $GATE failed $DIR/green — it fires on the fix too"
  exit 1
fi

echo "PROVEN: $GATE fails red ($red), passes green ($green)"

And run across every canary in the repository, which is what makes the proof continuous rather than a thing that happened once:

$ for d in tests/fixtures/*-canary/; do
>   gate=".claude-runtime/scripts/framework-verify-$(basename "$d" -canary).sh"
>   bash tests/run-canary.sh "$gate" "$d"
> done

PROVEN: framework-verify-fork-module-seam.sh fails red (1), passes green (0)
PROVEN: framework-verify-feature-module-path.sh fails red (1), passes green (0)
INERT:  framework-verify-store5.sh passed tests/fixtures/store5-canary/red — it detects nothing
PROVEN: framework-verify-test-output-layout.sh fails red (2), passes green (0)
...

The third line is the one worth having. That gate had been green in CI for months, and nothing about its output had ever been different from the gates that work.

The set -uo pipefail without -e is deliberate and is the detail that bites people: under set -e the first command is expected to fail, so the script exits before it can report anything, and a harness that dies on its own red run looks exactly like a harness that passed.

The over-broad check in the second assertion is the half that usually gets left out. A gate with an inverted condition fails on the red fixture for the right-looking reason and fails on the green one too, and a harness that only asserts "red fails" certifies it as working.

The third line matters more than it looks. A green fixture that differs from the red one in several ways cannot tell you which difference the gate responded to, and the pair is only evidence if the delta between them is the defect and nothing else.

The fourth is what protects against acquired inertness, which is the more common of the two failures in a codebase of any age. Gates do not usually ship broken; they ship working and then quietly lose contact with a renamed path, and nothing about the day that happens looks different from any other day.

What this does not require is a heavyweight framework. The fixtures can be directories with a few files in them and a script that runs the gate twice and compares exit codes. The discipline is in the ordering and the isolation, not in the tooling.

The cheapness matters for adoption. A fixture pair that takes ten minutes gets written; one that needs a harness gets deferred to a tidier week, and the gate ships unproven with a note about it that nobody reads afterwards.

How we proved it#

The ordering has caught gates that would not have worked, which is the entire argument and is hard to appreciate until it happens. Writing the fixture, running the gate, and seeing it pass when it should have failed is a specific and slightly unsettling experience, and it happens often enough that I no longer trust the other order.

The one-class rule came from a fixture that was doing nothing. A multi-violation fixture was reporting red on a gate that detected one of its three violations, and the two it missed had been "covered" for months. Splitting the fixture made that visible immediately.

Acquired inertness is the one I would watch for in an older codebase. The gates that worry me are not the new ones — those get written with their fixtures now — but the ones that have been green since before the directory layout changed, and whose last genuine detection nobody can date.

What I have not measured is how many gates in an existing codebase are inert. The honest answer for any gate without a red fixture is that it is unknown, and auditing them means building the missing fixtures — which is the work anyway, so I have done it opportunistically rather than as a sweep.

What I have not solved: fixtures for defects that need a lot of context. Some classes only appear in a realistic project — a build configuration, a device, a running service — and a fixture that reproduces them is expensive enough that I have sometimes shipped the gate without one and written down that I did.

And a limitation worth naming: a red fixture proves the gate detects that instance of the class. A gate can pass its fixture and still miss a variant that nobody thought of, and the fixture gives no coverage guarantee over the class — it establishes the gate is not inert, which is a much weaker claim than that it is complete.

The thing that reframed it was realising that a check is a measuring instrument and nobody trusts an instrument that has never been seen to move. We would not accept a thermometer that always reads the same thing on the grounds that the readings look plausible, and a gate that has only ever been green is in exactly that position.

Which also explains why the red run cannot be replaced by argument. The point is not to convince anybody the gate should work — the logic usually does look right — it is to observe the instrument responding to the thing it is supposed to respond to.

What this generalizes to#

An alarm that has never sounded has not been shown to work.

The shape recurs anywhere detection is involved: a backup nobody has restored from, a failover never exercised, a monitor whose alert path was never triggered, an exception handler on a branch that has never executed. Each is believed because it exists, and each is in an unknown state until something makes it fire.

So the question worth asking of any check is when did this last fail, and what happened. If the answer is "never", that is not reassurance — it is the absence of evidence either way, and the cheap way to get the evidence is to break something on purpose while nobody is depending on the answer.