I applied my fix to 234 files at once. It broke 233 of them.

TL;DR#

A mechanical sweep is not 234 small changes. It is one hypothesis, executed 234 times — and if the hypothesis is wrong, every single edit is wrong together.

Mine was wrong. The fix was correct in principle, and it referenced a variable that existed in 2 of the 234 files I applied it to. A passing test suite went from 65 green to 2 green in one command.

The rule is not "be careful with sed". It is: run it on one, prove it, then run it on the rest — and the one costs about ninety seconds.

The case this post is about#

A repository with 482 contract-test scripts, one per command. Each asserts its command's runtime documentation contains certain required clauses, by grepping a file.

A separate rule in the same repo requires that when a runtime exceeds 200 lines, a block of it moves out to a shared sub-module, leaving a pointer behind:

> Load layers/<area>/commands/_shared/<name>.md

Those two rules are in direct conflict. Obey the size rule and the clause moves one file along; the contract test still greps only the original file, goes red, and names a clause that is demonstrably still present. I had just hit this, fixed it properly for one command, and written a small resolver that reads a runtime plus whatever it loads.

Then I went to apply it everywhere. A survey said 249 scripts had the same latent problem and 32 of them — about one in eight — were already red because of it.

The symptom#

Before:

sample of 80:  65 pass · 15 fail

I wrote the insertion, ran it across 234 files, and re-measured:

after the sweep:  2 pass · 78 fail

The first failing script said this:

capsules/ai-agent/agent-sync/verify/verify.sh: line 27: ROOT: unbound variable

My inserted line referenced ${ROOT}. I had read one script — the one I fixed by hand — seen ROOT in it, and written the sweep against that memory.

A count across the files I had just edited:

CAP_DIR   228
FW_ROOT     2
FW          2
ROOT        1
R           1

ROOT existed in one of them. I had applied a line that could not possibly work to 233 files, in a second, and every one of them failed in exactly the same way.

Why the obvious fix failed#

"Read the code before you change it." I did read the code. I read one file, generalised from it, and never checked whether the generalisation held. Reading is not sampling, and a sample of one has no variance to show you.

"The fix was tested." It was — on the file I hand-fixed first, which passed. That file was the one with ROOT. My test case and my counter-example were the same file, and I had picked the only member of the population that worked.

"A dry run would have caught it." Probably, and I did not write one, because a dry run for a text insertion feels like ceremony. The honest version: a dry run would have shown me 234 plausible-looking diffs. What catches this is not previewing the edit, it is executing the result — the inserted line only fails when the script runs.

"Use a safer tool." The tool was fine. Nothing about sed versus a Python rewrite versus an editor macro changes whether ${ROOT} is defined in the target.

Every one of these treats the problem as the mechanics of editing. The problem was that I had one hypothesis — these scripts define ROOT — and I never tested it before betting 234 files on it.

The actual mechanism#

A sweep collapses N independent decisions into one. Editing 234 files by hand is 234 chances to notice. Editing them with one command is a single decision, made once, applied blind. The labour drops by two orders of magnitude and so does the number of opportunities to be wrong — which sounds like a benefit and is exactly the hazard.

Correct-in-principle and correct-in-this-file are different claims. My insertion was right about what needed to happen. It was wrong about the local fact that makes it executable. Sweeps are characteristically strong on the first and weak on the second, because the whole point of a sweep is that you stopped looking at individual files.

Homogeneity was assumed, never measured. The files were generated from a shared template, so I treated them as identical. They had drifted — five different names for the repository root, across 234 files that look the same from above. One grep -c would have told me. It took four seconds to run, afterwards.

And the part I find least comfortable: a guard tried to stop me and I went around it. Mutating those files through a shell bypassed seventeen registered pre-write hooks, because a file changed by a script never reaches the tooling that watches edits. The guard existed for exactly this, I did not notice I had evaded it, and nothing told me until a different guard fired later for an unrelated reason.

The fix#

One file, executed, measured — then the rest:

# 1. ONE file. Apply the change.
apply_fix capsules/client/kmp-client/verify/verify.sh

# 2. RUN IT. Not "does the diff look right" — does the result work.
bash capsules/client/kmp-client/verify/verify.sh   # FAIL -> PASS   ← the gate
                                                    # anything else: stop

# 3. Only now, measure the population you are about to touch.
for f in $TARGETS; do grep -ohE '^[A-Z_]+=.*(dirname|rev-parse)' "$f" | cut -d= -f1; done \
  | sort | uniq -c | sort -rn
#  228 CAP_DIR
#    2 FW_ROOT      ← five shapes the sweep would have to handle
#    2 FW
#    1 ROOT
#    1 R

# 4. Apply to the rest, then re-measure against the SAME baseline as step 0.

Step 2 is the one people skip, and it is the only step that can fail usefully. A diff review answers does this look like what I meant; running it answers does this work, and those come apart precisely when a local assumption is wrong.

Step 3 is the cheap one. Counting the population costs seconds and converts "they're all the same" from an assumption into a number. When I finally ran it, the answer was five shapes, and the right design was to need none of them — invoke by a path relative to the script rather than by whatever the file happens to call its root.

The second attempt went one file first. It failed too — and the failure was worth having, because it exposed a real bug in my resolver that would otherwise have shipped into every file: on macOS, mktemp requires the Xs at the end of the template, and mine read ...-XXXXXX.md. The first call created the file, every later call collided, and the variable holding the path came back empty. One file found that. Two hundred and thirty-four would have buried it in noise.

How we proved it#

By reverting and re-measuring, which is the only way to know a rollback actually rolled back:

baseline          65 pass · 15 fail
after sweep        2 pass · 78 fail
after revert      65 pass · 15 fail     ← identical to baseline, not merely "better"

That the post-revert numbers match the pre-sweep numbers exactly is the evidence. "It looks fine now" would not have been.

Then the corrected approach, one file at a time, each verified:

32 scripts targeted · 16 now pass · 16 still fail

And those 16 remaining failures are correct. They name clauses that genuinely do not exist anywhere — one wants a term absent from the runtime and every sub-module it loads. The fix removed the false failures and left the true ones, which is the outcome you want and is indistinguishable from "the fix did nothing" unless you check what the survivors are saying.

What this generalizes to#

Count the population before you treat it as one. "They're all generated from the same template" is a hypothesis with a number attached, and the number is one command away. Mine was 228 / 2 / 2 / 1 / 1 when I believed it was 234 / 0 / 0 / 0 / 0.

Run the result, do not review the diff. A diff shows intent. Execution shows truth. For a change that alters behaviour, the only test that counts is the one that runs it.

Your test case must not be your example. I fixed one file, used it to validate the sweep, and it happened to be the single file in the corpus where my assumption held. If the file you proved it on is the file you derived it from, you have proved nothing.

Revert is a measurement, not a hope. Compare to the baseline numerically. A rollback that leaves you at "about where we were" is a rollback you have not verified.

When a guard blocks a bulk operation, the guard is usually right. Mine was telling me the operation itself was unsanctioned, not that I had the syntax wrong. I read it as an obstacle. It was the most accurate feedback I got all day, and it arrived before the damage rather than after.

The thing that makes this failure mode durable is that the sweep feels like the careful option. You are being systematic; you are not hand-editing 234 files and missing one. But a systematic application of an untested premise is just a faster way to be wrong everywhere at once, and the ninety seconds it costs to try one file first is the cheapest insurance in software.

Related#

• 163 checks reported green. None of them read the published page. — the verification half of the same day
• Senior engineers decide without complete information — blast radius, priced before acting
• Most code review approves code. Here is review that improves it — why the diff was never going to show this

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.