A test that cannot fail is not a test. Mine passed twice.

I write a RED fixture for every gate: a deliberately broken input the check must reject. If the RED case passes, the gate is decorative.

Mine passed. Twice. And the reason is embarrassing enough to be worth writing down, because it is not specific to me.

TL;DR#

• A RED fixture that contains the token it is testing for will pass while proving nothing — including when the token sits in a comment explaining its own absence.
• The bug is a property of the FIXTURE, not the matcher, so tightening the assertion does not fix it.
• Every gate needs a proof it can fail: run the RED case and watch it go red before you trust the green.

The case this post is about#

One RED fixture, written to prove a gate could catch a missing rule, and one check:

grep -qiE "never loop|never .*batch" "$RUNTIME" \
  || fail "runtime does not forbid batching"

The fixture deliberately omitted the no-batch clause. The gate passed it. Twice.

It passed because the fixture described its own omission — a comment reading "the never loop clause DELIBERATELY OMITTED" contains the string never loop, so the grep matched the note explaining the absence rather than the rule itself.

This post follows that fixture, because a canary that cannot fail is worse than no canary: it is a passing check that certifies nothing, and every run of it increases confidence in a gate that has never been tested.

The symptom#

The gate asserts that a command's runtime declares a rule — that it must never loop an irreversible create. Implemented as a grep:

grep -qiE "never loop|never .*batch" "$RUNTIME" \
  || fail "runtime does not forbid batching"

The RED fixture is a copy of the runtime with that clause removed. Running the gate against it should fail. It passed.

Here is the fixture, and the bug is one line of it:

# RED canary fixture

Fails on the no-batch rule while the other checks hold. The regression it models
is the obvious convenience — "write and push all three" — added by someone who
never learned that the create cannot be undone.

<!-- the "never loop" clause DELIBERATELY OMITTED -->

Read the last line against the pattern. The grep looks for never loop. The comment explaining that never loop was removed contains the string never loop.

That is the whole defect. My editorial note about the omission quoted the exact tokens the check searches for, so the fixture asserted the presence of what it claimed to have deleted. The check was working perfectly; it found the phrase, because the phrase was there.

Why the obvious fix failed#

The first fix was to delete the explanatory comment. That worked, and it is not a fix — it is a patch on one instance of a general failure.

The general failure: a fixture written in the same language as the thing it tests will tend to mention what it removes. A RED case for a missing function signature will name the signature. A RED case for a missing header will explain which header is absent. Prose about an absence contains the absent thing, because that is how you write about it.

And the check is a substring match over the whole file. It cannot distinguish a normative statement from a description of one.

The second fix was to make the grep smarter — require the phrase at the start of a line, or inside a specific section. That helps a little and fails the same way eventually, because the fixture author is free to write anywhere and the matcher is guessing at structure.

The actual mechanism#

The reliable fix is a property of the fixture, not the matcher: a RED fixture must not contain the token it is testing for, in any form — including in a comment explaining its absence.

Written down in the fixture itself, where the next person will see it:

NOTE TO EDITORS: this fixture must not contain the guarded tokens in any form,
including in a comment describing their absence.

That is a real constraint and it costs something. You can no longer explain the omission in the obvious way. You write around it — describe the shape of the regression without quoting the phrase, or point at the green fixture for comparison.

The second half is a meta-check: prove the RED fixture is red before trusting a green result. A gate run is two runs, not one:

bash verify.sh                                      # real runtime  → must exit 0
RUNTIME=tests/canary/red/x.md bash verify.sh        # red fixture → must exit 1  preflight-ignore

If both exit 0, the gate is not checking anything, regardless of what the green run said.

The fix#

I now run every gate against every fixture and read which check fires, not just the exit code:

  green         PASS
  red           MP-3
  red-brief     MP-10
  red-manifest  MP-11

Each red fixture isolates exactly one check. That matters more than it first appears: when a fixture fails two checks, one of them is usually incidental, and an incidental failure hides a real pass. A fixture that fails MP-3 and MP-9 will still exit 1 after MP-3 is repaired and MP-9 breaks — so the exit code stays honest while the meaning quietly changes.

Adding a new check to a gate therefore means updating every fixture so it continues to fail only its own class. That is tedious and it is the job.

The harness that makes it tedious rather than impossible asserts the mapping both ways — every fixture fails its own class, and no fixture fails anybody else's:

# fixture -> the ONE check it must fail. green must fail nothing.
declare -A EXPECT=(
  [green]=""
  [red]="MP-3"
  [red-brief]="MP-10"
  [red-manifest]="MP-11"
)

verify_fixtures() {
  local gate="$1" dir="$2" failures=0

  for fixture in "${!EXPECT[@]}"; do
    local want="${EXPECT[$fixture]}" got
    # Capture WHICH checks fired, not just the exit code. The exit code is the
    # thing that was honest while the meaning changed underneath it.
    got=$(run_gate "$gate" "$dir/$fixture" | extract_check_ids | sort | join_commas)

    if [ "$got" = "$want" ]; then
      printf '  %-14s %s\n' "$fixture" "${want:-PASS}"
      continue
    fi

    failures=$((failures + 1))
    if [ -z "$want" ]; then
      printf '  %-14s EXPECTED PASS, fired: %s\n' "$fixture" "$got"
    elif [ -z "$got" ]; then
      # The defect this post is about: a RED fixture that passes.
      printf '  %-14s EXPECTED %s, fired NOTHING — this fixture proves nothing\n' "$fixture" "$want"
    else
      printf '  %-14s EXPECTED %s, fired %s — incidental failure masks the real one\n' "$fixture" "$want" "$got"
    fi
  done
  return $(( failures > 0 ))
}

The run that found it, and the run after the fixture was repaired:

$ verify_fixtures framework-verify-module-placement.sh tests/fixtures/module-placement-canary
  green          PASS
  red            EXPECTED MP-3, fired NOTHING — this fixture proves nothing
  red-brief      MP-10
  red-manifest   EXPECTED MP-11, fired MP-3,MP-11 — incidental failure masks the real one

$ verify_fixtures framework-verify-module-placement.sh tests/fixtures/module-placement-canary
  green          PASS
  red            MP-3
  red-brief      MP-10
  red-manifest   MP-11

Two different defects in the first run and the gate exited 1 for both of them. The red fixture quoted the phrase it claimed was missing, so nothing fired at all. And red-manifest failed MP-3 as well — which is the subtler one, because that fixture would keep exiting non-zero after MP-11 was deleted from the gate entirely.

How we proved it#

The check that found the original bug was running the RED fixture and reading the output rather than the status. Before:

$ RUNTIME=tests/canary/red/idea-medium-post.md bash verify.sh
idea-medium-post: PASS (MP-1..MP-9)

A RED fixture reporting PASS. After removing the offending comment:

$ RUNTIME=tests/canary/red/idea-medium-post.md bash verify.sh
FAIL: MP-3 runtime does not forbid batching
idea-medium-post: 1 failure(s)

Later the same day, a gate I had just written passed on the real runtime for a reason that had nothing to do with the rule. It required the words "provenance" and "procedural" — both of which already appeared in the file, incidentally, in unrelated sentences. Green, checking nothing. Tightened to require the literal field name and a specific phrase, it went red on the fixture as it should have from the start.

That is two instances in one day of the same class: a check that matches text present for unrelated reasons.

The shapes that keep recurring#

Having now been caught twice in a day, the pattern is narrow enough to list. A check is trivially-satisfiable when the thing it searches for can appear for a reason unrelated to the rule:

The self-documenting fixture. Prose about an absence contains the absent thing. Anything generated from a template that explains itself will do this.

The vocabulary match. Searching for a word the domain uses constantly. "Provenance" appears in any document about assets; "validate" appears in any document about input. If the token is common in the surrounding prose, the check is measuring topic, not compliance.

The substring accident. never .*batch matches across sentence boundaries. Regexes without anchors span more text than the author pictures, and the longer the file, the more chances there are for two unrelated fragments to satisfy one pattern.

The check that cannot see the state it cares about. Asserting that a step ran when what matters is what the step produced. I had a gate requiring a generator to be invoked; it passed happily while 303 of 304 outputs came from a fallback path, because invocation and outcome are different facts.

The defence is the same for all four: name the literal artifact, not the topic. A check for art_source: stitch cannot be satisfied by an essay about provenance. A check anchored to the start of a line cannot span two sentences. Specificity is what makes a check falsifiable, and falsifiable is the only kind worth running.

What this generalizes to#

Every assertion has a trivially-satisfiable version, and it is usually the first one you write. The defence is cheap: after writing a check, break the thing it guards and confirm it complains. If you cannot make it fail, you have not written a test — you have written a comment that runs.

The uncomfortable part is that this applies to the meta-check too. "Run the RED fixture and confirm it fails" is itself an assertion, and it can be satisfied by a fixture that fails for the wrong reason — which is precisely why reading which check fires matters more than reading the exit code. There is no bottom to this; there is only the point at which the evidence is specific enough that being wrong would require a coincidence rather than an oversight.

---

Related#

• A verification step that can pass while the bug is present is not a verification step — The same failure class, in the step that was supposed to catch it
• Ask a second agent to refute the first — What to do when a check agrees with you too easily

Keep reading#

One post a week on shipping software with AI agents — what broke, what the fix was, and the evidence. No digests, no course pitch. Subscribe.