I fixed 30 of 31 failures. Nineteen were left.

TL;DR#

A conformance checker reported 31 failures. I fixed 30 of them — every one verified, none reverted — and re-ran it.

Nineteen failures.

Nothing had regressed. The checker reports one problem per item, so what it had been showing me was never a defect count: it was the depth of a queue I could only see the front of. Clearing thirty front-of-queue problems exposed what had been standing behind them the whole time.

The number was the least informative thing in the output, and it was the only thing I had been tracking.

The case this post is about#

A repository with 482 components, each carrying a manifest and a contract test. A conformance script walks all of them and checks a handful of properties per component — required fields present, a declared verify script that exists and is executable, a test fixture directory, a content hash matching the body it describes.

The opening measurement:

FAIL: 31 capsule(s) non-conformant (scanned=482)

  20 × CF-1b   missing required frontmatter field (content_hash)
  10 × CF-1e   content_hash present but stale
   1 × CF-1c   verify script not executable

A clean, tractable list. Twenty missing a field, ten with a stale hash, one needing chmod +x. All mechanical, all fixable with the project's own sanctioned stamping tool.

So I fixed them. The chmod. The ten stale hashes, through the tool that computes them by the exact documented recipe. The twenty missing fields, added by hand and then stamped the same way.

Thirty of thirty-one, each one verified.

The symptom#

FAIL: 19 capsule(s) non-conformant (scanned=482)

   1 × CF-1b   missing required frontmatter field — the one I did not fix
  12 × CF-1c   verify script missing at the declared path
   6 × CF-1d   missing tests/ directory

Three things are wrong with that, and the first one is the one I noticed.

The total moved by twelve, not thirty. I had fixed thirty things.

CF-1e is gone entirely — all ten stale hashes, genuinely resolved. That part worked.

CF-1c went from one to twelve, and CF-1d from zero to six. Categories that were absent or negligible in the first report were now 18 of the 19, and the single survivor from the first report is the one component I had not fixed.

My first thought was that I had broken something. I had not: CF-1d had been true of those components all along. The checker had simply never got far enough to say so, because each of them was already failing an earlier check and the loop moved on.

Why the obvious fix failed#

"Something regressed — check what you changed." Nothing had. The components now reporting CF-1d were the same ones that had reported CF-1b, and the only edits they received were a frontmatter field and a hash. Neither can delete a test directory.

"The checker is flaky." It was perfectly deterministic. Run it twice on an unchanged tree and you get the same 31, then the same 19. Determinism and completeness are different properties, and I had been treating the first as evidence of the second.

"Count failures per check instead of per component." Closer, and still wrong — it would report the same numbers, because the checker never evaluates the later checks on a component that has already failed an earlier one. The data to aggregate differently does not exist.

"Just keep fixing until it reaches zero." That works, eventually, and it is why this is a reporting problem rather than a correctness one. But it makes the remaining work unknowable: at 31 I believed I was thirty-one fixes from done. I was not, and nothing in the output said so.

The actual mechanism#

The loop is the ordinary shape:

for component in all:
    for check in [CF-1a, CF-1b, CF-1c, CF-1d, CF-1e]:
        if not check(component):
            report(component, check)
            break          # ← the whole story is in this line

That break is a reasonable choice. Without it a badly-formed component emits five lines of cascading noise, most of them consequences of the first. With it, each component contributes at most one line and the report stays readable.

The cost is that the number it prints is the count of components with at least one problem, which is a different quantity from the number of problems — and the two are indistinguishable until you start fixing things.

Three consequences, in increasing order of how much they cost me:

Progress is invisible. I resolved thirty real defects and the headline moved from 31 to 19. A reasonable observer — including me, briefly — reads that as "mostly ineffective".

The composition changes under you. The failure mix went from 20/10/1 to 1/12/6. The second report is not a subset of the first; it is a different population reached by removing the first. Comparing them as though they were the same measurement is a category error, and the shared axis labels invite exactly that.

Depth is unknown and unbounded. After the second pass, I still cannot tell you how many problems exist in those 482 components. I can tell you 19 of them have at least one. There may be another layer behind CF-1d, and the only way to find out is to fix it and look again.

The deeper point is about what a short-circuiting check is for. As a work queue it is good: it tells you the next thing to do on each item, without drowning you. As a measurement it is actively misleading, because it looks exactly like a defect count — same units, same shape, same place in the output — and quietly answers a different question.

The fix#

Nothing about the checker needed changing. What needed changing was what I tracked:

# Not this — it conflates "items with a problem" with "problems".
FAIL: 31 capsule(s) non-conformant

# This. Same loop, honest headline.
FAIL: 31 of 482 capsules have at least one problem (first-failure only;
      later checks are NOT evaluated on a failing capsule — the true
      defect count is unknown and >= 31)

One sentence, and it would have saved me the ten minutes I spent looking for a regression that had not happened.

Where you genuinely need the count, the loop has to stop short-circuiting — and then you need the other half, or the noise returns. Both modes, in the checker itself:

type Check = { id: string; detail: string; passes: (c: Capsule) => boolean };

/** Declared order matters: an earlier failure is what hides the later checks. */
const CHECKS: Check[] = [
  { id: "CF-1a", detail: "frontmatter parses",         passes: (c) => c.frontmatter !== null },
  { id: "CF-1b", detail: "content_hash present",       passes: (c) => "content_hash" in c.fm },
  { id: "CF-1c", detail: "verify script exists +x",    passes: (c) => isExecutable(c.verifyPath) },
  { id: "CF-1d", detail: "tests/ directory present",   passes: (c) => c.hasTestsDir },
  { id: "CF-1e", detail: "content_hash matches body",  passes: (c) => c.fm.content_hash === sha256(c.body) },
];

type Mode = "queue" | "burndown";

function scan(capsules: Capsule[], mode: Mode) {
  const rows = capsules.map((c) => {
    const failed = [];
    for (const check of CHECKS) {
      if (check.passes(c)) continue;
      failed.push(check);
      // The `break` is CORRECT for a queue and is what makes the count a queue depth.
      // Keeping it means the headline below must say so.
      if (mode === "queue") break;
    }
    return { capsule: c, failed };
  }).filter((r) => r.failed.length > 0);

  if (mode === "queue") {
    // The honest headline. `>=` because the later checks were never evaluated on these.
    console.log(
      `FAIL: ${rows.length} of ${capsules.length} capsules have at least one problem ` +
        `(first-failure only; later checks NOT evaluated on a failing capsule — ` +
        `the true defect count is unknown and >= ${rows.length})`,
    );
  } else {
    // A real count, and grouped by check so one root cause does not read as five
    // independent defects — which is the noise the `break` existed to suppress.
    const total = rows.reduce((n, r) => n + r.failed.length, 0);
    const byCheck = new Map<string, number>();
    for (const r of rows) {
      for (const f of r.failed) byCheck.set(f.id, (byCheck.get(f.id) ?? 0) + 1);
    }
    console.log(`FAIL: ${total} problem(s) across ${rows.length} capsule(s), all checks evaluated`);
    for (const { id, detail } of CHECKS) {
      const n = byCheck.get(id);
      if (n) console.log(`  ${String(n).padStart(3)} × ${id}  ${detail}`);
    }
  }
  return rows;
}

Run in burndown mode, the same tree that reported 31 reports 54 problems across 31 capsules — which is the number I had believed I was tracking all along, and it was never printed. The two modes disagree by 23 on a tree where nothing has changed, and that gap is the whole post.

For a queue, keep the break and label the number honestly. For a burndown, drop the break and group the output. The mistake is wanting one number to serve both, which is what the original line was quietly trying to do.

How we proved it#

The claim to test was specific: the newly-visible failures were masked, not caused. "Nothing I did could have caused them" is an argument, not evidence.

So: take the components now reporting CF-1c, and check them against the list of components that had reported CF-1b in the first run.

CF-1c failures after the fix:        12
of those, present in the original CF-1b list:  12 of 12

Complete overlap. Every newly-visible failure came from a component whose earlier failure I had just resolved — which is what masking predicts and what "I broke something" does not.

The same test disconfirms the alternative cleanly: if my edits had caused them, I would expect at least some to appear on components I never touched. None did.

And one honest caveat I cannot resolve: this shows the 19 were previously masked. It does not show there is no third layer behind them. There may be. The only instrument I have reports the front of the queue.

What this generalizes to#

Ask what a failure count counts. "31 failures" and "31 items with at least one failure" print identically and mean different things. The difference is invisible until you fix something, and then it is expensive. Any tool that reports per-item rather than per-problem has this property — most linters with a fail-fast mode, most schema validators, most conformance walkers.

A changing composition is the tell. If the categories reshuffle between runs while the total barely moves, you are draining a queue, not a backlog. Watch the mix, not the number.

Deterministic is not complete. Running twice and getting the same answer tells you the tool is stable. It tells you nothing about whether the answer is the whole answer, and the two feel identical from the outside.

Prove masking by set overlap. When a fix "reveals" new problems, the cheap test is whether the newly-failing items are the ones you just touched. Complete overlap means masked; a spread across untouched items means caused. It takes one comm and settles an argument that otherwise runs on intuition.

Report what you could not see. The most valuable line a short-circuiting checker can print is the one admitting it stopped looking. Mine did not, and nothing about its output was wrong — it just answered a question I had not asked, in the units of the question I thought I had.

Related#

• 163 checks reported green. None of them read the published page. — a check asserting on the wrong surface
• A step that never ran is not a step that passed — the same confusion, one layer up
• Partial translation is a number, not a feeling — when the count is the honest instrument

Keep reading#

This is part of Evals & verification — the half of shipping that only shows up once something is live. Follow MobileByteSensei for the rest.