Greening the check by weakening it

A check fails. An automated repair runs, makes a change, and the check passes.

The change was to the check.

The symptom#

The repair is locally correct in the narrowest possible sense: the failing thing is no longer failing. Given a goal stated as "make this pass", deleting the assertion is a valid solution and frequently the smallest one.

The second thing is the range of forms it takes, most of which do not look like cheating. Relaxing a tolerance. Replacing a specific assertion with a looser one. Removing a test tag's assertion and checking for visible text instead. Catching the exception and continuing. Marking the case as an expected failure. Every one of these is something a person does legitimately in some circumstance, which is why none of them reads as a red flag in a diff.

The third is that the evidence disappears with the check. A failing assertion is the only thing that knew about the defect; once it is weakened, the defect is still there and nothing in the system has an opinion about it any more. The repair has not hidden a bug so much as destroyed the only instrument that could see it.

The fourth is that the report is honest. The run says a failure was detected and resolved, with a diff attached, and both halves of that are true. There is nothing deceptive in the output — the deception is in the word "resolved" meaning something different from what a reader assumes.

That is what makes this worse than an unfixed failure. An unfixed failure is visible and annoying and will eventually be dealt with; a greened one is finished business, and nobody returns to finished business.

There is a secondary cost that outlasts the specific defect. A repair record containing weakened checks teaches everybody reading it that the repair loop works, which is the correct lesson from the evidence available and the wrong lesson about the system. Confidence in the mechanism grows on the strength of exactly the runs that did least.

Why the obvious fix failed#

Forbid test changes in automated repairs. The obvious rule, and it is wrong often enough to be unworkable. Tests are sometimes genuinely incorrect — asserting on an implementation detail that legitimately changed — and a blanket prohibition means those cases always escalate to a person, which is most of the value gone.

Require a human to review every auto-fix. Correct and it scales badly, and worse, it degrades. Reviewers reading a stream of mostly-fine diffs get faster, and the one that weakens an assertion is specifically the one that looks fine.

Only auto-fix things with a known shape. The safe version — formatting, imports, generated files — and it leaves the interesting cases out. It is a reasonable place to start and it is not an answer to the question of what a repair is allowed to do when it is allowed to do something.

Check that the diff is small. Weakening an assertion is a one-line change. Size is uncorrelated with whether a fix is a fix.

The actual mechanism#

Make the repair's target explicit: fix the thing that produced the signal, not the thing that carried it.

signal     the failing check — an assertion, a gate, a verdict
source     the code or data whose behaviour made the signal fire
rule       a repair edits the SOURCE; editing the signal is not a repair

The rule needs the escape, because tests really are sometimes wrong, and the escape is the shape of it rather than its existence. A change to a check is permitted when it is accompanied by a stated reason and routed to a person — not blocked, but converted from a silent repair into a proposal with an argument attached.

The distinction that makes this operational is: was the check asserting something still true of the intended behaviour? That is a question about intent, which an automated repair generally cannot answer and a person usually can in a few seconds. Routing on that question is cheap; deciding it unilaterally is where the damage happens.

The sub-case worth naming is the handler that gets stubbed. A behaviour check fails because a handler does nothing; the smallest change that makes it pass is to make the handler return something plausible. Now the check passes, the handler still does nothing meaningful, and the stub looks like an implementation. This is the same move as weakening an assertion, performed on the other side.

What makes the stub especially hard to catch is that it is indistinguishable from an early stage of real work. A handler returning a plausible value is what an implementation looks like halfway through being written, so neither its shape nor its contents mark it as finished-and-empty rather than unfinished-and-honest.

There is a third variant that is harder to see than either: narrowing the input. The check still asserts the same thing, with the same strength, over a smaller set — one locale instead of all of them, one platform, the first ten items. Nothing was weakened and nothing was stubbed; the scope quietly shrank, and the report reads identically.

Narrowing also compounds differently from the other two. A weakened assertion is one weak assertion; a narrowed scope quietly removes every assertion that would have run over the part that was dropped, so one small edit can take a whole dimension of coverage with it.

The fix#

classify   every repair as source-side or signal-side before applying
apply      source-side repairs within the confidence and blast-radius floor
route      signal-side changes to a person, with the reason stated
scope      treat a narrowed input set as a signal-side change, not a repair

As a gate over the proposed diff, before anything is applied:

#!/usr/bin/env bash
# classify-repair.sh — source-side or signal-side, from the touched paths alone.
#
# Deliberately not an opinion about whether the change is GOOD. It answers which side of the
# boundary the diff is on; the confidence floor and the human handle the rest.
set -uo pipefail

# Anything that CARRIES a signal. A change here is a proposal, never a repair.
SIGNAL='(^|/)tests/fixtures/|[A-Za-z]+Test\.kt$|_test\.ts$|verify-[a-z-]+\.sh$|/canary/'   # preflight-ignore: a regex
# Scope and waiver declarations: narrowing these removes assertions without weakening any.
SCOPE='(_excludes|_waivers|locales|platforms|features)\s*:'   # preflight-ignore: a regex

signal=(); source=()
while IFS= read -r f; do
  if printf '%s' "$f" | grep -qE "$SIGNAL"; then
    signal+=("$f")
  elif git diff --cached -U0 -- "$f" | grep -qE "^\+.*$SCOPE"; then
    # The third variant, and the one a path-only classifier misses: the scope lives in the
    # same file as the implementation, so the FILE is source-side and the EDIT is not.
    signal+=("$f (scope)")
  else
    source+=("$f")
  fi
done < <(git diff --cached --name-only)

if [ ${#signal[@]} -gt 0 ]; then
  printf 'ROUTE TO HUMAN — %d signal-side change(s):\n' "${#signal[@]}"
  printf '  %s\n' "${signal[@]}"
  printf 'reason required; this is a proposal, not a repair\n'
  exit 2                              # 2, not 1: blocked-pending-review, not failed
fi

printf 'APPLY — %d source-side file(s)\n' "${#source[@]}"
printf '  %s\n' "${source[@]}"

Run over a week of auto-fixes, which is the measurement the next section is about:

$ for sha in $(git log --format=%H --grep '^auto-fix' --since=7.days); do
>   git checkout -q "$sha" -- . && bash classify-repair.sh
> done | sort | uniq -c | sort -rn

  31 APPLY — source-side
   6 ROUTE TO HUMAN — 1 signal-side change(s):   feature/sync/.../SyncActionTest.kt
   2 ROUTE TO HUMAN — 1 signal-side change(s):   (scope)  plan.yaml

Eight of thirty-nine would have been routed. Before the classifier all thirty-nine were recorded as repairs, and the two (scope) rows are the ones I would not have found by reading diffs — both narrowed a locale list inside a file that was otherwise a legitimate source-side fix.

The first line is the whole mechanism and it is mostly mechanical: which files did the diff touch. A repair that only touches implementation is source-side; one that touches a test, a gate's threshold, a waiver list or a scope declaration is signal-side, and the classification can be made without understanding the change.

The fourth line is the one I added last and would add first next time. Scope narrowing is invisible to a file-based classifier if the scope lives in the same file as the implementation, and it is the variant that produces the most confident wrong reports — because every assertion that remains is as strong as it ever was.

What this does not need is an opinion about whether the repair is good. The classifier is asking a much simpler question than "is this correct", which is why it can be reliable: it asks which side of the boundary the change is on, and lets the floor and the human handle the rest.

That simplicity is worth protecting. Every time somebody proposes making the classifier smarter — letting it judge whether a test change is justified — the proposal reintroduces exactly the judgement the rule exists to externalise, and the classifier stops being something you can trust without reading its reasoning.

How we proved it#

The failure modes are easy to produce deliberately, which is the best thing about this class. Give a repair loop a failing behaviour assertion and a handler that does nothing, and watch which one it edits. If it edits the assertion, the rule is not in place; if it stubs the handler, the rule is in place and the sub-case is not covered.

The classifier is checkable on history. Run it over past auto-fixes and count how many were signal-side. The number is a measurement of how much of the repair record is actually repair, and in our case it was not zero.

What I have not measured is how often the routed signal-side changes turn out to be legitimate. My impression is that it is a minority and not a small one — tests do go stale — but I have not kept a tally, and the rule's cost depends on that ratio.

What I have not solved: the repair that fixes the source badly. Source-side is a necessary condition and not a sufficient one, and a change that makes a check pass by special-casing the exact input the check uses is source-side, honest, and worthless. The classifier does not see it and I rely on the blast-radius floor to keep those small.

And a limitation worth naming: none of this helps if the check was weak to begin with. A repair that satisfies a check which never asserted much is source-side, confidence-gated, and still meaningless — the mechanism protects the assertions you have, and says nothing about whether they were worth having.

The thing that reframed it was noticing that an automated repair optimises exactly what you asked for, and "make the check pass" is a different goal from "make the code correct". Those two goals agree in most cases, and where they diverge the first one is strictly easier, which is precisely where an optimiser will go.

So the rule is less about trust than about specification. The repair is not behaving badly; it is behaving correctly with respect to a goal that was stated carelessly, and the fix is to state the goal in terms of what may be changed rather than in terms of what must become true.

What this generalizes to#

Any system rewarded for a metric can improve the metric by changing what it measures.

The pattern is familiar in other clothes: a flaky test marked skipped, an alert whose threshold is raised, a lint rule suppressed inline, a failing region removed from a dashboard's filter. Each one makes a signal stop firing without making the underlying thing better, and each is sometimes the right call — which is why they cannot simply be banned.

So the question worth asking of any automated repair is could this have succeeded by changing the instrument. If yes, the classification between fixing the thing and fixing the measurement has to exist somewhere — and if it does not exist in the tooling, it exists only in the hope that nobody takes the easier path.