A warning is not a refusal, and the driver could not tell

A batch of forty-nine items was running through a check before each one was created. It stopped at nineteen.

Nothing had failed. Item nineteen had produced a warning, and the driver reading the check's output could not distinguish a warning from a refusal.

The case this post is about#

One batch of forty-nine, which stopped at nineteen:

19 finding(s) · 0 blocking · 19 warning

The driver stopped because the check reported findings. The check was entirely correct — nineteen posts had a warning worth seeing, none of them was a reason to refuse anything, and the exit code was zero.

The driver was reading the finding count rather than the blocking count. So a check doing exactly its job halted thirty items of real work, and the fix is not in the check.

This post follows that one line, because the two numbers in it mean different things and the output puts them side by side in prose — which is readable by a person and ambiguous to anything else.

The symptom#

A halt with no failure behind it. The check's own report said zero blocking findings and one warning, which is a pass — the warning was a note about length, the sort of thing a person reads and decides about. The driver saw output that did not match what it expected and stopped.

The second thing is what the driver was actually keying on. It looked for a particular word in the output and treated its absence as a problem. That works for exactly as long as every passing run produces that word, and a passing run with a warning produces a different line — so the driver was reading a formatting detail as a verdict.

The third is that the halt was the right behaviour for the wrong reason, which made it hard to argue with. Stopping a batch when a check does not clearly pass is sound policy. The defect is that the check had clearly passed, and the driver had no way to know, so a correct policy was applied to a misread input.

The fourth is where the cost landed. Thirty items did not get created, which is recoverable, and the run reported a stop rather than a completion, which is honest. The real cost was the half hour spent looking for what was wrong with item nineteen — because the stop implied something was.

That is the general tax on an ambiguous halt. A stop is a claim that something needs attention, and a stop with no cause behind it sends somebody to look for one — which they will do thoroughly, because the alternative explanation is that the tooling is lying to them, and that is not where anybody starts.

Why the obvious fix failed#

Make the check not warn. The first instinct and it destroys the thing that makes warnings useful. A check that can only pass or fail has to decide, for every finding, whether it is worth stopping the world — and most findings are not, which is why the warning level exists.

Match more strings. Add the warning-shaped output to the patterns the driver accepts. This works until the next output format and it encodes the same mistake more thoroughly: the driver is still reading prose meant for a person and inferring a verdict from its shape.

Parse the output properly. Closer, and still reading the wrong artifact. Prose output is for humans; it is allowed to change wording, add context, reorder lines. A driver that parses it is coupled to a presentation layer and will break on an improvement.

Always continue on warnings. The opposite error, and worse. Some warnings genuinely should stop a batch, and a driver that ignores the level entirely has given up the ability to distinguish — which is the same defect pointing the other way.

Both errors come from the same place: the driver holding an opinion about severity. Whether that opinion is "stop on everything" or "stop on nothing", it has been formed somewhere that cannot see what the finding actually is, and no amount of care in choosing between the two makes that a good place to decide from.

The actual mechanism#

Have the check emit a machine-readable verdict separate from its human output, and gate on the count that means stop.

blocking    a count of findings that must prevent the next step
warning     a count of findings a person should see and may accept
exit code   non-zero if and only if blocking is non-zero
gate        the driver reads the blocking count, never the prose

The third line is the cheapest version and would have been enough on its own. An exit code is the one piece of a command's output that is unambiguously for a program, and keying the driver on it removes every question about wording. The reason it was not used is that the check's exit code did not distinguish the two levels at the time — which is the actual bug, one level deeper than where it appeared.

Which is worth noting as a diagnostic habit in itself. The thing that broke was the driver; the thing that needed fixing was the check's exit code. A fix applied where the symptom appeared would have produced a driver that parses prose more cleverly — more code, same coupling, and the real gap still there.

The sub-case worth deciding deliberately is which findings are blocking. That is a judgement and it belongs in the check rather than in the driver, because the check is where the domain knowledge is. A driver deciding that some warnings are serious enough to stop on has reimplemented the check's own classification, badly, somewhere nobody will look for it.

There is a related question about what the driver does with warnings it is not stopping for. The answer that has worked is to collect and report them at the end, as a list, so the run completes and the person sees everything at once — rather than surfacing each one at the moment it occurs, where it competes with the work in progress.

The ordering matters more than it sounds. A warning raised mid-run arrives while somebody is thinking about something else and is dismissed as noise; the same warning in a list at the end arrives when the work is done and attention is free. Same information, different odds of being acted on.

The fix#

emit       separate blocking and warning counts, in a stable machine-readable form
exit       non-zero on blocking, zero on warnings alone
gate       the driver on the exit code or the blocking count, never on prose
collect    warnings through the run, report them together at the end

The driver keys on a count, never on text — which is what makes a new warning category safe to add:

type Finding = { severity: "block" | "warn"; rule: string; detail: string };

export function report(findings: Finding[]) {
  const blocking = findings.filter((f) => f.severity === "block").length;
  const warnings = findings.filter((f) => f.severity === "warn").length;

  for (const f of findings) console.log(`  [${f.severity}] ${f.rule} — ${f.detail}`);
  console.log(`\n${findings.length} finding(s) · ${blocking} blocking · ${warnings} warning`);

  // The EXIT CODE is the contract. Prose is for humans; this is for the driver.
  return blocking > 0 ? 1 : 0;
}

# The consuming driver. Keying on `^ok ` or on the absence of the word "warning"
# is what halted a 49-post run at post 19 over a word-count advisory.
if verify "$post"; then
  publish "$post"
else
  echo "blocked: $post" >&2
fi

The two outcomes a text match conflates and an exit code does not:

$ verify posts/a
  [warn] length:below-target — 1714 words
1 finding(s) · 0 blocking · 1 warning
$ echo $?
0                                  # advisory — the run continues

$ verify posts/b
  [block] secret:vault-alias — possible vault alias in body
1 finding(s) · 1 blocking · 0 warning
$ echo $?
1                                  # refusal — the run stops

The first output contains the word "warning" and the string "finding". Any driver matching on either treats an advisory as a refusal, and the failure looks like the gate being strict rather than the driver being wrong.

The fourth line is what makes the third acceptable to everybody. A driver that continues past warnings feels careless unless the warnings arrive somewhere — and a single list at the end of a run is both more useful than nineteen interruptions and more likely to be read.

What makes this worth the small amount of work is the asymmetry of the failure. A driver that stops too eagerly costs a halted batch and an investigation into a non-problem; a driver that continues too eagerly costs whatever the blocking finding was going to prevent. Both are real, and the way to avoid choosing between them is to make the signal unambiguous rather than to tune the threshold.

Tuning is the tempting move because it feels like progress and produces a visible change. It is also endless: every adjustment trades one failure for the other, and the position of the dial is argued about indefinitely because neither setting is right while the signal is ambiguous.

How we proved it#

The numbers describe it exactly. Forty-nine items, halted at nineteen, zero blocking findings, one warning. Every one of those figures was in the output at the time, and the conclusion follows from putting them side by side — the check reported a pass and the driver stopped.

The behaviour is reproducible on demand. Run the check against an item that produces a warning and observe the driver's decision. It stops, every time, for a run the check considers passing, and that takes one invocation to demonstrate.

What I cannot tell you is whether this had happened before at a smaller scale. A batch of three that stopped at two would have looked like an ordinary interruption and been re-run without comment. Nothing recorded the reason for a stop in a form that can be searched, so earlier instances are not recoverable.

Which is its own small lesson: a stop that does not record why it stopped is an event you can only learn from once, at the moment somebody is watching. Recording the reason costs a line and turns a class of interruption into something you can count.

What I have not solved: the warnings themselves. Collecting them at the end is an improvement in delivery and does nothing about whether anybody acts on them. A list of nineteen warnings at the end of a successful run is, realistically, a list nobody reads — and I do not have a mechanism that makes a non-blocking finding reliably get attention without making it blocking.

And a limitation of gating on exit codes: it moves all the judgement into the check, which is correct and means the check's classification has to be right. A finding wrongly marked blocking now stops batches with full authority, and the appeal process is editing the check — which is a higher bar than it sounds when somebody is mid-run.

The practical mitigation is to make the classification easy to read and easy to change — a short list of what is blocking and why, kept next to the check rather than buried in its implementation. Then the appeal is a conversation about a visible list rather than an archaeology exercise.

The thing that reframed it was noticing the driver and the check disagreed about what the output was for. The check was writing for a person: a summary, in words, with a level. The driver was reading it as an interface. Neither was wrong about its own job, and the mismatch between those two readings is where the whole episode lived.

What this generalizes to#

Human-readable output is not an interface, and treating it as one creates a contract nobody agreed to.

The pattern is everywhere automation meets tools built for people. A script grepping log lines for a phrase. A pipeline keying on a command's stdout rather than its status. A monitor matching an error string that gets rephrased for clarity. A deployment checking for a word in a build summary. In every case the producer believes it is writing prose and is free to improve it, and the consumer believes it is reading a contract.

So the question worth asking of any automated decision is what exactly is it keying on, and did anybody promise that would stay the same — because if the answer to the second half is no, the automation is depending on a sentence somebody is going to reword.

The cheapest protection is for the producing tool to offer an explicit machine surface — an exit code, a status field, a structured line — and say so. Then prose is free to improve, consumers have something stable to key on, and the question of what counts as a contract has an answer rather than a convention.