The running app and the spec disagree, and only one is documented

A generated file has a bug. The fix is three characters, it is obvious, and editing the file takes about four seconds.

Nobody will ever know it happened, including the person who regenerates that file next month.

The case this post is about#

One hand edit to one generated file — the fastest fix available, and the only one with no record:

export async function verifyGenerated(file: string) {
  const src = await Deno.readTextFile(file);
  const stamp = /\/\/ generated-digest: ([0-9a-f]{64}) from (\S+)/.exec(src);
  if (!stamp) throw new Error(`${file}: no generated-digest stamp — is it generated at all?`);

The edit was correct. It fixed a real bug, it shipped, and it was gone at the next regeneration — along with the bug fix — because the generator rewrote the file from a spec that had never been changed.

Nothing reported its absence. This post follows that one edit, because the file it lived in looks ordinary, the diff looked like a normal fix, and the thing that distinguishes a generated file from a written one has to be in the file or nobody will know which they are editing.

The symptom#

Two things now describe the same behaviour and they disagree. The specification says one thing; the source that actually runs says another; and the difference exists only in a working tree, where it looks exactly like ordinary code.

The second thing is the lifespan. The edit is correct, it ships, it works — and it has an expiry date nobody wrote down, which is the next time anything regenerates that file. At that moment the fix vanishes, the bug returns, and the regeneration looks completely routine because from its own point of view it is.

The third is that the returning bug is almost impossible to attribute. It was fixed, there is a commit saying so, the fix is no longer present, and nothing in between looks like a deletion. The most common outcome is that somebody fixes it again the same way, which resets the clock and guarantees a third occurrence.

The fourth is scale. One hand edit is a small liability. A file that has been hand-edited for a year is a file nobody dares regenerate, which means the generator is now effectively disabled for that file, and the specification it reads from has become decorative without anybody deciding that.

That last state is the one worth avoiding, because it is terminal. Once regeneration is unsafe the whole mechanism stops paying for itself: the spec is maintained out of habit, the source is maintained by hand, and the project is carrying the cost of both with the benefits of neither.

There is a milder version that is almost as bad, because it looks like a decision rather than a drift: the file is left alone and marked somehow — a note, a ticket, a convention that this one is special now. That records the state without resolving it, and a year later nobody can say whether the file is special because of a real constraint or because somebody was once in a hurry.

Why the obvious fix failed#

A comment at the top saying do not edit. Universal and ignored, for a good reason: the person editing has a real bug and the comment offers no alternative. A prohibition without a path is a prohibition that loses to urgency every time.

Make generated files read-only. Better, and it converts the problem into a permissions error at the worst possible moment. Somebody in a hurry chmods the file, makes the edit, and now the drift exists and the guard has been proven ignorable, which is worse than not having it.

Check the generated files into a separate directory. Keeps them visually distinct and does nothing about the edit — it just makes the hand-edited file easier to spot in review, which is a detection improvement rather than a prevention one, and only if somebody reviews.

Regenerate on every build. Would catch it immediately, by destroying the edit the moment it was made. In practice this is only tolerable when generation is fast and deterministic, and when it is not, the check gets relaxed to "regenerate sometimes", which is the original problem with extra scheduling.

There is a fifth that gets proposed and is worth answering because it sounds principled: stop checking generated files into version control at all. It removes the file somebody could edit, and it also removes every property that made the generated code reviewable — you can no longer see what a spec change did, or bisect a regression through it, or read the thing that actually runs. It trades a drift problem for a visibility problem and most teams discover the trade the hard way.

The actual mechanism#

Make the generated artifact detectably unmodified, and make the upstream path short enough that nobody wants the shortcut.

mark      every generated file carries a digest of its own content
check     a gate recomputes it; a modified file fails, by name
route     the gate's message names the spec file to edit instead
cost      regeneration for one file is seconds, not a whole-project run

The first two lines are the detection and they are the easy part. The mechanism that actually changes behaviour is the fourth: hand edits happen because the correct path is slow. If fixing the spec and regenerating one file takes two minutes, almost nobody edits the output; if it takes twenty, the prohibition is a wish.

The third line matters more than it looks. A gate that says "this file was modified" produces annoyance; a gate that says "this file is generated from that spec, edit there and run this" produces the behaviour you wanted. The difference between those two messages is most of the adoption.

The sub-case that keeps the whole thing honest is the edit that cannot be expressed upstream. It exists: a case the generator does not handle, a workaround for a library bug, something genuinely one-off. Pretending otherwise produces a gate people disable. The answer is a sanctioned, visible form — a region of the file explicitly marked as hand-owned and preserved across regeneration, or a declared override file — so the exception is recorded rather than smuggled.

Hand-owned regions have their own failure mode, which is worth naming before anybody adopts them: they grow. A file with one preserved block is a file with an escape hatch; a file with eleven is a hand-written file wearing a generator's header. The honest guard is to count them and treat growth as a signal that the generator is missing a case, rather than as evidence the escape hatch is working.

The other thing a counted escape gives you is a worklist. Each preserved region is a case the generator could not express, written down by the person who hit it, at the moment they hit it — which is a far better description of what the generator is missing than anything anybody would produce by sitting down to think about it.

The fix#

digest    stamp each generated file with a hash of what was generated
verify    fail the build on any generated file whose content has moved
name      report the spec that produces it and the command that regenerates
escape    an explicit hand-owned region, preserved and counted

The verifier and the preserved region are one mechanism — the regions must survive regeneration, so the generator and the checker read the same markers:

const OWNED = /^\s*\/\/ <hand-owned:(\w+)>$([\s\S]*?)^\s*\/\/ <\/hand-owned:\1>$/gm;
const stripOwned = (src: string) => src.replace(OWNED, "");

export async function verifyGenerated(file: string) {
  const src = await Deno.readTextFile(file);
  const stamp = /\/\/ generated-digest: ([0-9a-f]{64}) from (\S+)/.exec(src);
  if (!stamp) throw new Error(`${file}: no generated-digest stamp — is it generated at all?`);

  // Hash WITHOUT the hand-owned regions: those are expected to differ, by design.
  const got = await sha256(stripOwned(src));
  if (got === stamp[1]) return { ok: true, owned: [...src.matchAll(OWNED)].length };

  throw new Error(
    `${file} was edited by hand.\n` +
    `  it is generated from ${stamp[2]}\n` +
    `  regenerate:  deno task gen ${stamp[2]}\n` +
    `  to keep an edit here, wrap it:  // <hand-owned:reason> … // </hand-owned:reason>`,
  );
}

The message is what changes behaviour — a bare digest mismatch is an obstacle, this is an instruction:

$ deno task verify-generated
✗ core/store/ProfileStore.kt was edited by hand.
    it is generated from design/profile/data-flow.yaml
    regenerate:  deno task gen design/profile/data-flow.yaml
    to keep an edit here, wrap it:  // <hand-owned:reason> … // </hand-owned:reason>

$ deno task verify-generated          # after wrapping the workaround
ok · 42 files · 3 hand-owned regions

Three is the number to watch. One region is an escape hatch; eleven is a hand-written file wearing a generator's header, and the count is what tells you which one you have.

The second line has to fail rather than warn, for the usual reason: the edit it is catching produces working code, so there is no other pressure to deal with it, and a warning on a passing build is read by nobody.

The fourth line is the one that determines whether the first three survive six months. Without a sanctioned exception, every genuinely-necessary hand edit becomes an argument about the gate, and gates lose those arguments eventually — usually at the worst moment, when something is broken and the rule is between somebody and a fix.

What the escape must not be is silent. A preserved region that nobody counts is indistinguishable from the drift this whole mechanism exists to prevent; the counting is what keeps it an exception rather than a migration path back to hand-written files.

The counting also wants a direction. The number going up is not automatically bad — a generator meeting new cases will accumulate a few — but it should go down sometimes, and a number that has only ever risen is a generator nobody is improving, with the escape hatch doing the work instead.

How we proved it#

The disappearing fix is reproducible in two commands. Edit a generated file, regenerate it, and diff: the edit is gone, no warning, exit zero. Watching that happen is more convincing than any argument about discipline, because the surprise is how completely ordinary the regeneration looks.

The digest check catches it immediately and names the file, which is the part that makes it actionable. A hand-edited file reports itself at the next build rather than at the next regeneration — days earlier, while the person who made the edit still remembers why.

The routing message is the part I would not skip. The first version of this check reported a digest mismatch and a path, which is technically complete and practically useless to somebody who did not know the file was generated. Adding the spec path and the exact regeneration command turned the gate from an obstacle into an instruction, and that is the change that stopped people asking how to switch it off.

What I have not measured is how many hand edits existed before the check. The drift that had accumulated was found file by file as the digests were introduced, and I did not keep a count — so I can tell you the mechanism works and not how big the problem had been.

What I have not solved: the generator that is simply wrong. A hand edit is sometimes a correct response to a generator producing bad output, and routing it upstream means fixing the generator — which is a bigger job than the three-character fix and lands on somebody who may not own that code. The gate converts a silent local fix into an explicit upstream task, and does not make the upstream task smaller.

And a limitation worth stating: a digest proves the file has not been edited since generation. It says nothing about whether the generated output was right, and a file that is faithfully generated from a wrong spec passes every check here. This mechanism maintains agreement between two artifacts; it has no opinion about whether either is correct.

The thing that reframed it was realising the hand edit is not the problem — it is a symptom with a cause that is almost always the same: the upstream path was slower than the deadline. Every one of these I have looked at came down to regeneration being expensive, or the spec being hard to find, or the generator being someone else's.

Which means the enforcement and the ergonomics are the same work. A gate added to a project where regeneration is slow just relocates the pain; a gate added to a project where regeneration is two minutes is barely noticed, because the behaviour it demands is already the path of least resistance. I would do the second part first.

What this generalizes to#

Anything with an upstream has a shortcut that works, and the shortcut is invisible precisely because it works.

The pattern is the same for a patched vendored dependency, a configuration edited on the server, a schema changed in the database rather than in a migration, a tuned value set in a running process. In every case the local change is correct, immediate, and uncaptured — and the system's own normal operation is what erases it.

So the question to ask of any derived artifact is what happens to a change made directly to it. If the answer is "it works, and the next regeneration removes it", you do not have a discipline problem, you have an unmarked trap — and the people who fall into it will be the ones who were trying to fix something quickly.