A step that never ran is not a step that passed

A pipeline reports green. Every check that ran, passed. Four checks did not run at all — the session was interrupted, a device was unavailable, a prerequisite was never satisfied.

Nothing was wrong with any of them. They simply never happened, and the report had no way to say so.

The case this post is about#

One interrupted run, and the four verdicts a completion check needs in order to describe it:

passed       ran and succeeded
failed       ran and did not succeed
not-run      did not execute — blocks completion, is not a failure
excluded     explicitly declared inapplicable, with a stated reason

The run implemented 4 of 12 features and stopped. Verification never ran, the device test never ran, and the release step never ran — not failed, simply never reached.

A check with two states has to put those three somewhere. Calling them failures sends somebody to debug working code; calling them passes declares the project done. The structural verdict said done.

This post follows that one run, because the state the check was missing is the state the world was actually in.

The symptom#

A completion claim that is true about everything it mentions. The steps that executed succeeded; the summary lists them; the number is correct. The steps that never started contribute nothing to the output, because there is nothing for them to contribute — a step that did not run produces no result, and an absent result is not a failing result.

The second thing is that the gap is created by ordinary circumstance rather than by error. A run interrupted and resumed. A capability that needs hardware nobody has plugged in today. A stage whose prerequisite was quietly skipped. None of these is a bug, and all of them produce the same silence.

The third is that the binary shape of most status reporting forces the collapse. A field that can hold pass or fail has no value for "did not happen", so whoever wrote it had to choose, and the choice almost always lands on treating absence as non-failure — which is reasonable locally and means a pipeline can report complete while a third of it never executed.

The fourth is where it surfaces. Not during the run and not in review, but later, when somebody asks whether a particular thing was verified and the honest answer turns out to be that nobody knows. The report says green, which is evidence about the steps that ran and silence about the rest, and silence is indistinguishable from assent in a two-state field.

The delay is what makes this expensive rather than merely untidy. By the time the question is asked, the run is weeks old, the conditions that caused the skip are gone, and the only way to find out is to run the whole thing again — which is a much larger ask than it would have been at the time, when the gap could have been closed by plugging something in.

Why the obvious fix failed#

Make skipped steps fail. The blunt version, and it conflates two genuinely different things. A step that cannot run because the hardware is absent is not the same as a step that ran and found a problem, and treating them identically makes the signal useless on any day somebody is working without a device.

Require everything to run. Correct in principle and unworkable in practice. Some capabilities legitimately do not apply to some projects, and a rule that every stage must execute everywhere turns into a thicket of exceptions that are themselves unaudited.

Note the skips in the log. They are in the log. The log is not what anybody reads when deciding whether something is done, and a line saying a stage was skipped has no effect on a summary that says complete.

Check manually before declaring done. Works exactly once, performed by the person who knows to look. It is the kind of discipline that holds while somebody remembers the incident that caused it and decays over the following months.

The actual mechanism#

Give the status field three values, and make the third one block.

passed       ran and succeeded
failed       ran and did not succeed
not-run      did not execute — blocks completion, is not a failure
excluded     explicitly declared inapplicable, with a stated reason

The third line is the whole change and the fourth is what makes it survivable. Without a sanctioned way to say "this genuinely does not apply here", a blocking not-run is intolerable and somebody will route around it; with one, the route around is a declaration somebody wrote and can be read.

The important property of the exclusion is that it is explicit and attributable. A capability marked inapplicable with a reason is a decision; a capability silently absent is an accident. Both produce a green run, and only one of them can be reviewed — which is the entire difference.

It is also the difference between a system that accumulates understood exceptions and one that accumulates unexplained ones. The first grows a list somebody can audit; the second grows a set of behaviours nobody can account for, and both look identical from the outside until somebody asks.

The sub-case that matters is where the list of applicable steps comes from. If it is derived from what ran, the check is circular and every run is complete by construction. The set of capabilities that should have run has to be enumerated independently — from a registry, from the project's own declaration — and compared against what did.

The circularity is easy to introduce without noticing, because deriving the list from the run is the convenient implementation — the runner already knows what it dispatched. That list answers "what did I do" and the question is "what should I have done", and those only coincide when nothing was skipped, which is precisely the case the check exists to detect.

The fix#

enumerate   the applicable capabilities independently of the run
record      per capability: passed, failed, not-run, or excluded
block       completion on any not-run, as firmly as on a failure
require     a reason on every exclusion, and surface the list

The second line is worth doing even before the blocking. Simply recording the three states, without changing what the pipeline permits, produces a report that distinguishes "verified" from "not examined" — and that distinction is most of the value, because the decision somebody was about to

make on a green report changes once they can see the shape of it.

The enumeration and the verdict, in full. The registry is the independent source — deriving the list from what ran is the circularity the post is about:

type Verdict = "passed" | "failed" | "not-run" | "excluded";

export function completion(
  registry: string[],                       // what COULD apply — independent of this run
  ran: Record<string, "passed" | "failed">, // what the run actually produced
  excludes: Record<string, string>,         // explicit opt-outs, each with a reason
) {
  const rows = new Map<string, { verdict: Verdict; why: string }>();

  for (const cap of registry) {
    if (cap in ran) rows.set(cap, { verdict: ran[cap], why: "" });
    else if (cap in excludes) rows.set(cap, { verdict: "excluded", why: excludes[cap] });
    else rows.set(cap, { verdict: "not-run", why: "never executed" });
  }

  const notRun = [...rows].filter(([, r]) => r.verdict === "not-run").map(([c]) => c);
  const failed = [...rows].filter(([, r]) => r.verdict === "failed").map(([c]) => c);

  // not-run BLOCKS, exactly as a failure does, and is reported as its own thing.
  return { rows, complete: notRun.length === 0 && failed.length === 0, notRun, failed };
}

On a run that was interrupted after four of twelve capabilities:

$ pipeline report
  passed    8
  failed    0
  excluded  0
  not-run   4   device-test · idea-verify · idea-release · kmp-implement
complete: false

The old two-state version printed passed 8 · failed 0 and called it green. make on a green report changes once they can see the shape of it.

What makes the enumeration affordable is that the list of capabilities is small and changes slowly. It is a dozen entries in a registry, not a computed thing, and keeping it accurate is a matter of adding a row when a capability is added — which is exactly the moment somebody is thinking about it.

Where it does go stale is in the other direction: a capability that is retired and left in the registry blocks forever on something that no longer exists. That is a loud failure rather than a silent one, which is the right way round, and it does mean the registry needs an occasional read rather than only ever being appended to.

The enumeration also has to be scoped, which is the part I got wrong first. A global list of every capability the system knows about, applied to every project, produces a wall of exclusions that nobody reads — and a list of exclusions nobody reads is the silent gap again, wearing a declaration. The list that works is the one derived per project from what that project actually is, so the exclusions it carries are the few genuinely interesting ones.

How we proved it#

The failure is demonstrable by interrupting a run. Stop a pipeline partway, resume it, and ask what it reports. If the answer is complete, every stage after the interruption is being treated as having succeeded, and that takes one deliberate interruption to establish.

The three-state version changes the conversation immediately. A report showing eight passed and four not-run is read differently from one showing eight passed, even though the underlying facts are identical. Nothing about the system changed; what changed is that the absence became visible.

What I cannot tell you is how many past runs were affected. Historical reports recorded two states, so a past green cannot be re-examined to find out which stages actually ran. Everything before the change is unresolvable, which is the usual cost of adding a distinction late.

What I have not solved: the exclusion that is wrong. A capability marked inapplicable with a plausible reason passes every check in this scheme, and nothing verifies that the reason is true. It converts a silent gap into a stated one, which is a real improvement and is not verification — somebody still has to read the exclusions occasionally and ask whether they are still correct.

And a limitation of blocking on not-run: it makes interrupted work louder, which is correct and is also friction on a perfectly normal day. A run that cannot finish because a device is unplugged now refuses to report done, and the person who simply wanted to see how the rest went has to read past a halt. I think that is the right trade and it is a trade.

The softening that has worked is to make the halt informative rather than merely obstructive: report everything that did run, in full, and then state plainly which stages did not and why. Somebody who wanted to see the rest still sees the rest; what they do not get is permission to call it done.

The thing that reframed it was noticing that two-state fields silently encode a policy. A field that can hold pass or fail has already decided what absence means, and nobody made that decision explicitly — it fell out of the data type. That is a surprisingly common way for consequential behaviour to enter a system: not through a choice, but through a shape that had no room for a third case.

Once you start looking, the shapes are everywhere — a nullable field standing in for three conditions, an enum that predates a case, a return type that cannot express "did not attempt". Each is a decision made by whoever picked the type, usually before the third case existed, and inherited by everybody since as though it were deliberate.

What follows from that is a small habit rather than a large principle: when adding a status field, write down the states the thing can actually be in before choosing the type, including the ones that feel like non-events. "Has not been attempted" always feels like a non-event while you are designing, because at design time nothing has been attempted yet and the case has no weight. It acquires weight the first time a report is read by somebody making a decision.

What this generalizes to#

A two-state field forces a third state to masquerade as one of the other two, and it will choose the quieter one.

The pattern is everywhere. A boolean "verified" that cannot distinguish unverified from not-applicable. A test result with pass and fail and no concept of not-collected. A permission check that treats an absent rule as allow. A health endpoint that reports healthy because nothing reported unhealthy. In every case the system is answering a question it was never asked, using a default nobody chose.

So the question worth asking of any status is what are the real states of this thing, and does the field have room for all of them — because if it does not, one of them is being silently reclassified, and it is almost always being reclassified as fine.