The audit was green, specific, and about the wrong artifact

An audit ran over the built site and reported every page clean. It named a count: a hundred and thirteen pages, all passing.

The source directory held a hundred and forty-five. The build it had inspected was two days old.

The case this post is about#

One quality gate, reporting a clean pass over 113 pages, while the source held 145.

Every check it ran was correct. Every page it examined really was clean. It was reading a build directory from two days earlier, because the audit ran before the build in that particular path, and nothing in its output said which build it had looked at.

The number 113 is the tell and it is not in the report — the report says clean, and a reader has no denominator to compare it against. A gate with no count and no build identity cannot distinguish "audited everything" from "audited a directory nobody rebuilt".

This post follows that one number, because the pass was honest about what it examined and silent about what it did not.

The symptom#

A green report with a number in it that nobody checked against anything. The count looked like evidence of thoroughness — it is more convincing than a bare "passed", because it demonstrates the tool went and counted something. What it demonstrates is that the tool counted what it found, which is a different claim.

The second thing is that the check itself was correct. Every one of those hundred and thirteen pages genuinely was clean. The tool had no defect, its rules were right, and its output was accurate about the thing it looked at. The failure is entirely in which artifact it was pointed at.

The third is that nothing in the pipeline makes staleness visible. A built directory looks the same whether it was produced a minute ago or a week ago; it has files, they parse, they are complete. There is no property of a stale build that distinguishes it from a current one short of comparing it against something.

The fourth is the shape of the gap, which is the part that makes this worth writing down. The missing thirty-two pages were the newest ones — everything added since the last build. So the audit was passing precisely on the material that had been reviewed before, and silent on all the material that was new. It was checking the part least likely to be wrong.

That inversion is what makes staleness worse than an outright broken check. A check that is simply down produces no report and somebody notices the silence. A check reading a stale snapshot produces a report that is maximally reassuring about exactly the material nobody has looked at — it is not merely useless, it actively covers the riskiest ground.

Why the obvious fix failed#

Run the audit in CI after the build. The right structure and it is what made this invisible locally. The pipeline order was correct in CI; the failure happened when the audit ran on a developer machine against whatever build happened to be lying there.

Which means the bug was never in the code or the configuration. It lived in the gap between two ways of invoking the same tool — one thought about carefully, one typed from memory a few hundred times.

Check that the build directory exists. It existed. Existence is the check everybody writes and it answers the wrong question — the problem was never a missing build, it was an old one, which passes an existence check with no hesitation.

Existence checks are everywhere for a reason: they are the first thing anybody writes, they catch a real failure, and they cost one line. The trouble is that having written one, the directory now feels checked — and the next question, whether what is in it corresponds to anything current, does not get asked, because a check already exists there.

Build before auditing, every time. Correct and expensive enough to get skipped. A full build to run a thirty-second audit inverts the cost, and the first time somebody is in a hurry they will run the audit alone and get a green answer that means nothing.

Trust the count. The count is what gave it away eventually, and only because somebody happened to know roughly how many posts there were. A number is only a check if something compares it to another number.

Until then it is decoration that reads as rigour, which is worse than no number at all. A report saying "passed" invites the question "on what"; a report saying "113 passed" answers a question nobody asked and quietly suppresses the one that mattered.

The actual mechanism#

Have the audit establish that its input is current, and refuse rather than report when it cannot.

count      the source artifacts and the built artifacts, and compare
compare    the newest source timestamp against the build timestamp
refuse     if the build is older than any source file
report     the two counts in the output, always, even when they agree

The first line is the cheapest and would have caught this immediately. A hundred and thirteen against a hundred and forty-five is not a subtle discrepancy, and the comparison requires nothing the tool did not already have — it was already walking one directory and could walk the other.

The fourth line is the one I would press hardest, because it changes what a green report means. An output that says "113 built, 113 source, all clean" is a different artifact from one that says "113 clean". The first is checkable by a reader at a glance; the second asks them to trust that the denominator was right, which is exactly the trust that failed.

Printing the denominator has a second effect that is harder to measure and probably larger: it changes what the person reading the report is doing. A bare pass invites acceptance; two numbers invite a comparison, and somebody who has compared two numbers once will notice the next time they disagree.

The sub-case worth separating is what counts as newer. Timestamps are the obvious comparison and they are weak: a checkout can rewrite them, a copy can preserve them, and clock differences across machines are real. Comparing counts is cruder and more robust; comparing content hashes is exact and costs more. I use counts as the first gate because it caught this one and would catch most of the class.

The fix#

compare    source count against built count, before auditing anything
refuse     a build older than its newest source, with both values named
print      denominator and numerator in every report, not just the result
locate     the audit downstream of the build in every path, not only in CI

The comparison is two counts taken from different places, and the refusal names both — a single number has no way to be wrong out loud:

export async function assertBuildCurrent(srcGlob: string, distDir: string) {
  const sources = await count(srcGlob);                       // from source
  const manifest = JSON.parse(                                 // from the build
    await Deno.readTextFile(`${distDir}/build-manifest.json`).catch(() => "null"));

  if (!manifest) {
    throw new Error(`no build manifest in ${distDir} — auditing a directory nobody built`);
  }
  if (manifest.pages !== sources) {
    throw new Error(
      `build is stale: ${manifest.pages} pages built, ${sources} sources on disk\n` +
      `  built at ${manifest.builtAt}\n` +
      `  run the generate step before auditing — the audit is reading an old build`);
  }
  return { sources, built: manifest.pages };
}

// Every report prints BOTH numbers, pass or fail. A result with no denominator
// cannot be checked by the person reading it.
export const report = (r: { sources: number; built: number; findings: number }) =>
  `${r.findings} findings over ${r.built}/${r.sources} pages`;

The audit that started this, before and after:

$ audit
  0 findings over 199 pages          # looked perfect

$ audit                               # with the comparison
build is stale: 199 pages built, 253 sources on disk
  built at 2026-09-28T14:02:11Z
  run the generate step before auditing — the audit is reading an old build

Fifty-four pages had never been built, so the audit could not have had findings about them. 0 findings was true and about a corpus two thirds the size of the real one.

The fourth line is where the real defect lived. The pipeline was correct and the developer path was not, and the developer path is where the audit is actually run most often — which is a general hazard: the carefully-ordered automated sequence and the ad-hoc local one are different pipelines, and only one of them gets designed.

The asymmetry is worth taking seriously because the undesigned path is usually the busier one. The automated pipeline runs on merges; the local one runs every few minutes while somebody is working, which is where almost all of the actual invocations happen and where none of the ordering guarantees were written down.

What makes the comparison affordable is that counting a directory is free next to everything else these tools do. The reason it was absent is not cost; it is that nobody had articulated "the input might be stale" as a failure mode, so there was nothing to write a check against.

That is the general reason cheap checks go missing. Nobody declines to write them on grounds of cost; they are absent because the failure they would catch had not been named, and an unnamed failure mode has no check-shaped hole for anybody to notice.

How we proved it#

The two numbers settle it on their own. A hundred and thirteen inspected, a hundred and forty-five present. No interpretation, no further investigation — the moment both numbers are in the same place, the conclusion is immediate, and the entire difficulty of this bug was that they had never been in the same place.

The missing set is exactly the newest material, which is checkable by sorting the difference by date. That is worth confirming rather than assuming, because it determines how bad the failure is: a random thirty-two would be an inconvenience, and the newest thirty-two means the audit was systematically blind to everything unreviewed.

What I cannot tell you is how many previous green reports were also stale. The audit kept no record of what it inspected, so earlier runs cannot be re-examined. The honest position is that this was found once and may have been happening for an unknown period before anybody put the two numbers together.

What I have not solved: the general case of a tool inspecting a derived artifact. Everything above is specific to one audit and one build. A test suite running against a stale compiled output, a linter reading a generated file, a report built from an exported dataset — each has the same shape and each would need its own currency check, and I have not gone looking for the others.

Writing that down is partly a note to myself, because the search is obvious and bounded: every tool in the repository that reads from a directory it does not produce. That is a list I could generate in an afternoon and have not, which puts it in the same category as every other backfill in this publication.

And a limitation of count-comparison: it catches additions and deletions and is blind to modification. A source file edited after the build produces identical counts and a stale audit, and nothing above would notice. Counts are a cheap first gate rather than a sufficient one, and I have described them as the former deliberately.

The upgrade path is clear when counts stop being enough: compare a hash of the source set against one recorded in the build. That is exact, catches modification as well as addition, and costs a pass over the files — cheap in absolute terms, and worth deferring until the crude check has stopped finding things.

The thing that reframed it was noticing that I had been treating "the check passed" and "the thing is good" as the same statement, when they are separated by a question nobody asks: what did the check actually look at. Every verification has an input, that input has a provenance, and a green result tells you about the input rather than about the world.

What this generalizes to#

A correct check on a stale input produces a confident wrong answer, and the confidence comes from the correctness.

The pattern recurs wherever verification happens at a distance from the thing being verified. A security scan against a container image built last month. A performance test against a deployment that was rolled back. Documentation coverage computed from a cached index. A dashboard aggregating a table that stopped being written to. In each case the measurement is accurate, the subject is wrong, and nothing in the output distinguishes the two situations.

So the question worth asking of any passing check is what exactly did it inspect, and how do I know that was current — because a check that cannot answer the second half is not reporting on your system. It is reporting on a photograph of your system, taken at a time nobody recorded.

The last clause is the one that bites. A photograph with a timestamp is evidence; a photograph without one is an anecdote — and almost every derived artifact in a build directory is the second kind unless somebody deliberately made it the first.