An agent-driven refactor, with the receipts

A refactor is a change that alters structure and not behaviour. The word does a lot of quiet work there, because nothing in the diff tells you which half you got.

So the useful question is not whether the agent did it correctly. It is what artifact you can point at afterwards that proves it.

The case this post is about#

One refactor: 36 feature-module include lines moved out of a template-derived Gradle settings file and into the fork's own seam, so a template sync could no longer delete them.

The claim is that nothing changed. The receipt is the system's own model of itself, captured before and after and diffed:

capture() {
  ./gradlew projects --quiet            > "$1/projects.txt"
  ./gradlew :app:dependencies --quiet   > "$1/deps.txt"
  find . -name '*.kt' | sort | xargs shasum -a 256 > "$1/sources.txt"
}

$ ./receipt.sh move-feature-modules
INERT — graph and dependencies identical
      72        # source files whose hash changed, all named in the migration plan

The word refactor claims the structure moved and the behaviour did not. Nothing in the diff separates those two halves, which is why the interesting artifact is not the diff.

The symptom#

The migration was mundane: thirty-nine module declarations lived in a build file that a sync script overwrites wholesale, and they needed to move to a companion file the sync leaves alone.

The stakes were specific. The file they were in gets replaced from upstream, so a declaration living there is one sync away from silent deletion — the build breaks and no file anyone owns shows a diff. Measured before the move: 39 real module includes in the synced file, of which 36 were locally added and 3 genuinely belonged upstream.

That is an ideal agent task — mechanical, repetitive, well-specified. It is also the exact shape where review stops working, because the diff is thirty-nine near-identical lines moving between two files and a reviewer's eyes slide off it. Reading it tells you the lines moved. It does not tell you the build still resolves the same graph.

Why the obvious fix failed#

The obvious check is "does it still build". It passes trivially and proves much less than it appears: a build succeeding after a move tells you the graph is valid, not that it is the same. Drop a module nobody currently depends on and the build is perfectly happy.

The second attempt is to diff the two files and confirm the lines match. This catches transcription errors and misses everything structural — ordering effects, a declaration that resolves differently from its new location, a duplicate that now shadows.

The third is to count. Thirty-nine before, thirty-nine after. Better, and still a proxy: it confirms nothing was dropped and says nothing about whether each one still resolves to what it did.

The actual mechanism#

Compare the resolved output, not the input.

For this migration that meant dumping the build system's own project graph before and after, and requiring the two to be byte-identical. Not similar, not equivalent-looking — identical.

That receipt has a property the others lack: it is produced by the thing whose behaviour is in question. The build system resolves the graph, so its own rendering of that graph is the closest thing to ground truth available, and a byte comparison of two renderings admits no judgement.

The constraint that made it interesting: the comparison had to happen without restarting the build daemon. A cold restart costs 15–30 minutes and is separately forbidden here, so "just tear it down and rebuild clean" was not on the table. The receipt had to be gathered from a warm system, twice, around the change.

The fix#

Three rules that generalise past this one migration.

The receipt is an artifact, not a claim. Two files and a comparison, reproducible by anyone. Not "I verified the graph is unchanged" — the two dumps, kept, and the command that says they match.

Prefer the output of the system under question. A linter's opinion about your build file is a

weaker witness than the build system's own resolved model, so the receipt asks the thing that actually decides:

# receipt.sh — capture the system's OWN model, before and after, and diff it.
capture() {
  ./gradlew projects --quiet            > "$1/projects.txt"
  ./gradlew :app:dependencies --quiet   > "$1/deps.txt"
  find . -name '*.kt' | sort | xargs shasum -a 256 > "$1/sources.txt"
}

mkdir -p .receipt/before .receipt/after
capture .receipt/before
run_migration "$@"
capture .receipt/after

# The claim is "inert": the project graph and resolved dependencies are unchanged,
# and ONLY the files the migration names have moved.
if diff -q .receipt/before/projects.txt .receipt/after/projects.txt >/dev/null \
&& diff -q .receipt/before/deps.txt     .receipt/after/deps.txt     >/dev/null; then
  echo "INERT — graph and dependencies identical"
  diff .receipt/before/sources.txt .receipt/after/sources.txt | grep '^[<>]' | wc -l
else
  echo "NOT INERT — the migration changed the resolved model:"
  diff .receipt/before/projects.txt .receipt/after/projects.txt || true
  exit 1
fi

The receipt for the migration that prompted this:

$ ./receipt.sh move-feature-modules
INERT — graph and dependencies identical
      72        # source files whose hash changed, all named in the migration plan

Both dumps stay in .receipt/. The claim is not "I checked" — it is two files and the command that compares them, which anybody can re-run.

The choice of what to capture is most of the design, and the rule that produced it is to ask the system under question rather than a tool that has an opinion about it. A linter can tell you a build file looks equivalent; the build system can tell you its resolved model is identical, and only one of those is the thing that will actually run. The same logic picks dependencies over reading version catalogues, and a hash of every source file over a count of changed files.

The second rule is that a receipt has to be cheap enough to produce on every attempt, or it becomes something you generate once to settle an argument. Two Gradle invocations and a find is seconds, which means it runs on the failed attempts too — and the failed attempts are where it earns its keep, because "NOT INERT" with a diff attached tells you exactly which assumption was wrong.

What a receipt cannot do is tell you the migration was a good idea. It establishes that the change was inert with respect to the model, which is a narrow and checkable claim. Whether the resulting layout is better is a judgement, and keeping those two things separate is what makes the narrow claim worth anything — a receipt that also tried to argue for the design would be an opinion with a checksum attached.

The failure it actually caught was a module that no longer appeared in the project list, because a settings include had been moved into a file the sync overwrites. The build still succeeded, which is the part worth sitting with: a valid graph is not the same as the same graph, and only the comparison distinguishes them.

That is the case I would hand to anyone sceptical of the whole idea. The build passing is the evidence most people would have accepted, and it was true, and the migration had still dropped a module. No amount of care in reading the diff would have caught it, because the diff was exactly what the plan described — the loss was in a file the plan did not mention and the sync owns.

The general form, if you take nothing else: when an agent reports that a change is safe, ask what artifact it would show you. "I checked" is a claim about a process. Two dumps and a comparison is a claim about the system, and the second one survives being wrong about the first.

It also changes how the work feels to supervise. Reviewing an agent's reasoning about a large mechanical change is slow and not very reliable — the reasoning is usually plausible and the failure is usually somewhere it did not think to look. Reviewing a receipt is fast and reliable, because the question collapses from "was this done correctly" to "do these two files match". second opinion. The build system's own resolved graph is the thing itself.

If you cannot produce the receipt, it is not a refactor. That is the line I find most useful. A change you cannot prove inert is a behavioural change you have not characterised, and it should be reviewed, tested and shipped as one — with the risk stated rather than assumed away by the word "refactor".

How we proved it#

The measurement before the change, which is what justified doing it at all:

real module includes in the synced file   39
of those, locally added                   36
in the companion file the sync preserves   0

Thirty-six declarations one upstream sync away from disappearing, and nothing in the fork's own files would have shown the loss.

And the receipt after: the project graph, dumped before and after, byte-identical. The move changed where thirty-six declarations live and changed nothing about what the build resolves.

What the agent got wrong#

Three things, and the pattern in them is the useful part.

It wanted to move all thirty-nine. Three of those declarations are genuinely upstream-owned — the sync re-supplies them, so moving them would have created duplicates on the next sync. The distinction is not visible in the file; it lives in knowing which modules the template ships. The agent had the file and not the provenance.

It proposed verifying by inspection. The first plan was to read both files and confirm the lines matched, which is the check that cannot see a resolution change. This is the failure this whole article is about, and it was the default proposal.

It reached for a clean rebuild to compare. Correct instinct, wrong cost — and blocked by a guard, which is the only reason it did not happen. The warm comparison was available and needed one more step of thought.

None of those is a mistake about the code. All three are mistakes about what would count as evidence, which is consistently where I find agent-driven changes weakest: the work is right and the proof is a restatement of the work.

The honest limitation: a byte-identical graph proves the build resolves the same modules. It does not prove the sync behaves as expected — that the preserved file really is preserved on the next upstream pull. That is a claim about a future event, and the only receipt for it is the next sync. I have written down what to check when it happens, which is weaker than a proof and better than an assumption.

Where this leaves review#

The thing I have actually changed is what I ask for in review of an agent-written refactor.

Not "does this look right" — thirty-nine moved lines always look right. Instead: what is the artifact that proves this is inert, and can I run the comparison myself? If the answer is a pair of dumps and a command, the review is thirty seconds and conclusive. If the answer is a description of care taken, the review is an hour and inconclusive.

That reframing also changes what the agent is asked to produce. "Move these declarations" invites a diff. "Move these declarations and give me the before/after graph" invites a diff and the receipt, and the second half is the part I would not have got by asking more carefully.

What this generalizes to#

When a change is supposed to alter structure without altering behaviour, decide in advance what artifact will prove the second half — and prefer something emitted by the system whose behaviour is in question over anything a reader or a linter concludes.

Then treat the absence of that artifact as information. A refactor you cannot prove inert is not a refactor with a missing test; it is a behavioural change wearing a safer word, and the honest move is to rename it and review it accordingly.