A stale doc does more damage than a missing one

// STITCH IS THE DEFAULT. The six procedural variants remain as the FALLBACK
// for a post generated without network or quota. They are not the preferred
// path any more.

Every word of that comment was false. The flag it describes was opt-in, and it threw on every real input — it had never once run to completion. 303 of 304 assets came from the "fallback" which was, in fact, the only path that worked.

The case this post is about#

One comment, at the top of a generator:

// STITCH IS THE DEFAULT. The six procedural variants remain as the FALLBACK
// for a post generated without network or quota. They are not the preferred
// path any more.

Confident, specific, and describing a code path that had never once run to completion. The default it names threw on any image large enough to be worth embedding, so 303 of 304 assets came from the "fallback".

The comment was true when it was written. Nothing about reading it reveals that it has stopped being true, because it is a claim about the present tense with no date and no measurement attached.

This post follows that one comment, because it is worse than no comment: it is the thing a reader consults to decide which path they are looking at, and it sent me the wrong way for a week.

The symptom#

A missing doc produces a question. A wrong doc produces a confident wrong answer, and the difference is that one of them ends the investigation.

Three from the same codebase, in one day:

The comment above. It stopped me looking. I was auditing why assets did not match their declared source, read that comment, and concluded the wiring was fine and the problem lay elsewhere. It cost an hour of looking in the wrong place.

A file header listing three operations when the file had four. The fourth — added that morning, by me — was undocumented, so a later reader would not know it existed. Harmless, until the missing one is the one they need.

A session-handling section describing a vault-based flow as the primary path, written before we discovered that path is blocked at the network edge and cannot work at all. It describes a design, accurately, that reality had already rejected.

All three were true when written. None announced that they had stopped being true.

Why the obvious fix failed#

"Keep docs updated" is not a mechanism, and the reason is worth being precise about: the person who breaks a doc is almost never editing the doc.

I broke the first comment by leaving --art opt-in while the comment said default. I was working on the compositing logic, forty lines away. Nothing I touched was the comment. There is no moment in that workflow where a careful person would have reread it.

The second failure mode is sharper still: the comment was never true. --art threw on any image over a few kilobytes, so at the moment it was written — describing intended behaviour — it was already describing something that could not happen. Aspirational documentation is indistinguishable from accurate documentation, right up until someone depends on it.

Review does not catch this either. A reviewer reads the diff. The comment is not in the diff.

The actual mechanism#

What worked was making the document's claim machine-checkable, so the check fails when the claim stops being true.

That does not mean testing your prose. It means identifying the specific factual assertion inside it and pinning that:

The third is the interesting shape — a check that fails on the presence of something. The claim is "we never do X", and the only way to verify it is to assert X does not appear.

My first attempt at one of these taught me the real lesson. I wrote a gate for the provenance claim that searched for the words "provenance" and "procedural". It passed immediately — both words already appeared in the file, in unrelated sentences. A check for a topic is not a check for a claim. Tightened to require the literal string art_source: stitch, it went red on a fixture as it should have from the start.

The fix#

Three habits, in order of how much work they are.

Record the measurement, not the conclusion. "Stitch is the default" ages badly. "Measured 2026-09-29: 303 of 304 assets were procedural" cannot age, because it is a dated observation rather than a claim about the present. If it is stale a reader can see that it is stale, which is the property the confident version lacks entirely.

Put the failure in the comment. The rewritten version of that header is longer and more useful:

// Chunked, because the obvious one-liner cannot encode a real image.
// btoa(String.fromCharCode(...bytes)) spreads every byte as an argument —
// 168,334 of them for the first cover — and dies with "Maximum call stack
// size exceeded". So the --art path threw on ANY image big enough to be
// worth embedding, which is why no Stitch cover ever shipped.

Nobody is tempted to "simplify" that back. The comment contains the reason it exists, and a reader who removes the chunking now knows exactly what they are reinstating.

Delete rather than leave. For the header listing three of four operations, the fix was not to list four — it was to note that the list is not authoritative and point at the dispatcher. A list that must be maintained in two places will diverge; the only question is when.

The first habit is the only one a machine can help with, and it can: a dated claim is checkable and an undated one is not.

type Claim = { file: string; line: number; text: string };

const DATED = /\b(measured|observed|as of|counted)\s+(\d{4}-\d{2}-\d{2})\b/i;
const ASSERTIVE = /\b(is|are|always|never|the default|every|all)\b/i;

function audit(claims: Claim[], today: string, maxAgeDays: number) {
  const out: string[] = [];
  for (const c of claims) {
    const dated = DATED.exec(c.text);
    if (dated) {
      const ageDays = Math.floor((Date.parse(today) - Date.parse(dated[2])) / 86_400_000);
      // A stale DATED claim is still true — it is a true statement about a past
      // measurement. It needs re-measuring, not correcting.
      if (ageDays > maxAgeDays) {
        out.push(`${c.file}:${c.line} measurement is ${ageDays}d old — re-measure`);
      }
      continue;
    }
    // An undated assertive claim cannot be checked at all, which is the problem:
    // nothing about reading it reveals whether it is still so.
    if (ASSERTIVE.test(c.text)) {
      out.push(`${c.file}:${c.line} undated claim about the present: "${c.text.trim()}"`);
    }
  }
  return out;
}

The two versions of the same comment, audited on the day the stale one was found:

audit([
  { file: "store-assets-render.ts", line: 12, text: "// Stitch is the default path for covers." },
  { file: "store-assets-render.ts", line: 12, text: "// Measured 2026-09-29: 303 of 304 assets were procedural." },
  { file: "idea-deploy.md", line: 88, text: "// All four operations are listed below." },
], today = "2026-10-07", maxAgeDays = 90).forEach((l) => console.log(l));

store-assets-render.ts:12 undated claim about the present: "// Stitch is the default path for covers."
idea-deploy.md:88 undated claim about the present: "// All four operations are listed below."

The dated version produces no finding, because it is eight days old and still true as a measurement. That is the whole asymmetry: the audit can tell you a dated claim needs re-checking and when, and about the undated one it can only tell you that it is unverifiable — which is also the most it could ever tell you, and the reason the first habit is worth the extra eight words at the keyboard.

How we proved it#

The direct test is whether a gate catches the doc going stale, so I broke each one deliberately.

Reverting the create path to the importer, against the gate that pins "create uses the API":

FAIL: runtime does not use the API path (deno task publish)
FAIL: runtime does not name the publication the create must attach to

And the negative-assertion gate, given a runtime that reaches for the forbidden operation:

FAIL: runtime reaches for the publish op — publishing is manual by operator decision

The honest counter-example is the one I cannot gate. The file header listing three operations instead of four is prose about structure, and no cheap check distinguishes a complete list from an incomplete one. I fixed it by hand and it will go stale again. Some documentation is genuinely un-pinnable, and the useful response is to keep it thin rather than pretend a process will maintain it.

The failure that is hardest to see#

There is a third category, worse than both, and I only noticed it because it happened twice.

A doc can be stale about a capability that never worked. Not "this used to be true", but "this was written from the design and the implementation never matched". The animation pipeline is the cleanest example: a runtime declared animated assets mandatory, named the producer, and described the output format. All accurate as a specification. The artifact that producer emits — CSS keyframes in an HTML file — could not be rendered by either surface it was destined for. Zero animations had ever been seen by a reader, and the documentation said the capability was required.

That shape defeats every check I have. A gate can verify the runtime names the producer. It cannot verify the output is useful, because "useful" is about a rendering surface the gate never sees. The doc and the gate agreed with each other perfectly while both were disconnected from reality.

What surfaced it was a count. Not a review, not a test — someone asking "how many animations do we actually have?" and the answer being zero. The same question found the provenance gap an hour earlier: 303 of 304 assets came from a path the docs called a fallback.

So the cheapest audit I know is to take each capability a document claims and ask how many times has this actually happened. A capability with a count of zero is either unused or broken, and the documentation will not distinguish those for you — it describes both the same way.

What this generalizes to#

Absent documentation makes a reader look. Wrong documentation makes them stop, which is why it costs more. When you write a claim about how the system behaves, ask what would have to be true for it to be false, and whether anything would notice — then either pin that with a check, or write the dated measurement instead of the timeless assertion, or accept the claim will rot and keep it short enough to be cheap when it does.