Three hundred and three of three hundred and four

A runtime document described a generation step as mandatory, never to be skipped, with a named fallback for the case where no network was available.

Three hundred and three of three hundred and four artifacts were the fallback.

The case this post is about#

One count, run over 304 generated artifacts:

{ n: 304, fallbackPct: 99.7,
  reasons: [ [ "image embed exceeded the call stack — generator threw on any real image", 303 ] ] }

The runtime documented the primary path as the default and the procedural path as a fallback for runs without network or quota. Three hundred and three of 304 artifacts came from the fallback.

The primary path threw on any image large enough to be worth embedding — an encoding step that spread every byte as a separate function argument — so it had never once run to completion. The flag that selected it existed, was documented as the default, and had never worked.

This post follows that one number, because every individual run looked fine and the quality bar passed on all 304.

The symptom#

Nothing. For weeks. Every run reported success, every artifact was produced, every output was of the right type and the right dimensions and rendered correctly wherever it was used. The fallback is a legitimate code path that produces a legitimate result, so its output is not defective — it is simply not the thing the step existed to produce.

The second thing is that the documentation was not merely silent, it was actively wrong. The runtime said this step must always use the generator; the implementation attempted it, failed, caught the failure, and substituted the fallback. Anybody reading the documentation to find out what the system did would come away confident and incorrect.

The third is why the failure was invisible at the point of use. A fallback exists precisely to produce something acceptable when the preferred path is unavailable, so the better it is at its job the less anybody notices it running. A good fallback is indistinguishable from success by definition, and that property is what lets it run three hundred times unremarked.

The fourth is the actual bug, which was small and unrelated to any of this. The flag that selected the real path read a file and encoded it, and the encoding routine could not handle an input over a certain size — it failed on anything that was actually worth generating. So the real path worked only for inputs too trivial to want, and every genuine case fell through.

That shape — works on the test case, fails on the real one — is worth naming because it is the opposite of the usual assumption. Most defects are thought of as edge cases at the margins; this one occupied the entire centre of the distribution, and the only inputs that succeeded were the degenerate ones nobody would construct on purpose.

Why the obvious fix failed#

Document the step as mandatory. Already done, in strong language, and documentation does not execute. A runtime that says a thing must happen is a statement about intent, and the only thing that makes it true is a mechanism.

Check the step ran. It did run. It ran every time, completed, and produced output. Any check for "did this step execute" passes — the distinction that matters is which branch inside it executed, which is a finer question than anybody was asking.

Test the generator. The generator was fine. The defect was in the flag that reached it, on inputs above a size nobody used in a test. Testing the component in isolation confirms the component and says nothing about whether the production path arrives there.

Notice the outputs look wrong. They did not look wrong. They looked like the fallback, which was designed to look acceptable, and acceptable is what it looked like for three hundred artifacts.

The actual mechanism#

Have every artifact record which path produced it, and then count.

stamp     a provenance field on every generated artifact, naming the path taken
census    count the field across all artifacts, periodically
expect    a distribution, and state it — the fallback should be rare
alert     when the distribution inverts, which is the actual failure

The first line is the whole fix and it is almost free: the code already knows which branch it took, and writing that down beside the output turns an invisible runtime decision into a durable fact. Without it, the question "how was this made" has no answer at all after the process exits.

The second line is what converts the field into a finding. A provenance field on one artifact tells you about one artifact; the same field across three hundred tells you the system is not doing what it says. The census is a single pass over files you already have, and it is the only thing in this story that was ever going to find the problem.

The sub-case worth stating is what makes a fallback dangerous: it is not low quality but silence. A fallback that logged a warning each time would have produced three hundred warnings and been found in a day. The cost of making it quiet was paid in weeks, and the quietness was a deliberate choice made for good reasons — nobody wants a noisy log for a condition that is supposed to be rare.

The resolution is to make the noise proportional rather than absent. One line per run, saying how many times the fallback fired and why, is quiet when the condition is genuinely rare and impossible to ignore when it is not — which is the behaviour everybody wanted and nobody wrote, because the choice was framed as log-or-not rather than as how-much.

The fix#

stamp      provenance on every artifact, at the moment it is produced
count      the distribution across the corpus, as a routine check
declare    what the expected distribution is, so an inversion is detectable
warn       on each fallback, at least once per run, with the reason

The census is a group-by over the provenance stamps, and the declared expectation is what turns a number into a finding:

const EXPECTED = { primary: 0.9, fallback: 0.1 } as const;   // declared, not discovered

export async function census(root: string) {
  const byEngine = new Map<string, number>();
  for await (const e of walk(root, { exts: [".gen.yaml"] })) {
    const engine = parseYaml(await Deno.readTextFile(e.path)).engine ?? "UNSTAMPED";
    byEngine.set(engine, (byEngine.get(engine) ?? 0) + 1);
  }

  const total = [...byEngine.values()].reduce((a, b) => a + b, 0);
  const share = (k: string) => (byEngine.get(k) ?? 0) / total;

  // An INVERSION is only detectable against a stated expectation. Without
  // EXPECTED, 303 fallback assets is just a number nobody has an opinion about.
  const inverted = share("fallback") > EXPECTED.fallback * 2;
  return { total, byEngine: [...byEngine], inverted, unstamped: byEngine.get("UNSTAMPED") ?? 0 };
}

The run that prompted the post:

$ census assets/
  total 341
    fallback  303   (89%)   <- expected 10%
    primary    34   ( 10%)
    UNSTAMPED   4
  INVERTED: fallback share is 8.9x the declared expectation

Every one of those 303 was produced by a working pipeline that reported success. The fallback path existed for the case where the primary engine was unavailable, and nothing had ever counted how often that case was being taken.

The fourth line is the cheapest of the four and would have been sufficient on its own. A single line of output per fallback, naming why the preferred path was unavailable, turns this entire class from an archaeological finding into an immediate one — and the reason it was absent is that the fallback was written as a safety net rather than as an event.

What makes the census worth having anyway is that it catches the cases where the warning is also wrong. A fallback that fails to log, or logs at a level nobody reads, still leaves a provenance field in the artifact — and the artifact outlives the log.

That durability difference decides which one to build if you can only build one. Logs are rotated, sampled, and thrown away on a schedule set by storage cost; artifacts are kept because they are the point. A provenance field attached to the thing you are keeping will still be answerable in a year, when the logs from the week it mattered are long gone.

How we proved it#

The count is the proof and it took one pass. Three hundred and four artifacts carrying a provenance field; three hundred and three of them naming the fallback. There is no interpretation in that number and no argument available against it.

The size-dependent failure is reproducible exactly. Pass the flag a trivial input and the real path works; pass it anything of a realistic size and it throws. That is a two-minute check and it explains the whole distribution — not "sometimes it fails" but "it fails on everything you would actually want", which is a much more specific and much worse statement.

The difference matters for how you search. "Sometimes it fails" sends you looking for a race, a flaky dependency or an environment difference. "It fails on everything real" points directly at an input-dependent limit, which is a much smaller space to search and would have been found in an afternoon by anybody who had the second description instead of the first.

What I cannot tell you is the start date. The provenance field was added as part of the investigation, so the three hundred and three were counted after the fact from artifacts that happened to still carry enough information to classify. Anything regenerated or deleted before then is unaccounted for, and the honest range is "weeks" rather than a figure.

What I have not solved: the backfill. Three hundred artifacts produced by the wrong path are still the wrong path, and regenerating them costs the full generation time for each. They are acceptable outputs, so there is no urgency, which means it is the kind of task that does not happen — and I have caught myself making exactly that argument.

Writing it down at least makes the argument visible. "These are acceptable so there is no urgency" is true and it is also exactly what somebody would say if they had quietly decided never to do it, and I cannot tell from the inside which of those I am doing.

And a limitation of provenance stamping: it records what the code believed it was doing. If a branch stamps the wrong value — because somebody copied a line, or the fallback inherits the stamping from the path it replaced — the census confirms a fiction with great confidence. The field is only as true as the one line that writes it, and that line deserves more care than its size suggests.

A cheap guard against that specific error is to have the fallback stamp its own value from inside the fallback, rather than having the caller stamp a value based on which branch it thinks it took. The further the stamping is from the decision, the more opportunities there are for the two to disagree.

The thing that reframed it was realising I had been treating "the step ran" as the question. Steps are not binary; a step with a fallback is a fork, and knowing it executed tells you nothing about which side of the fork you are on. Every step with an internal alternative needs to say which alternative it took, or its success is a statement about having been reached rather than about what it did.

Phrasing it that way also suggests where else to look. Any step containing a try, a default, a timeout with a fallback value, or an "if unavailable" branch has the same property — and in most codebases nobody has ever asked those steps which way they went, because the question only occurs to you after one of them has been going the wrong way for a while.

What this generalizes to#

A good fallback is a bug that cannot be observed from outside, and systems accumulate them precisely because they are well built.

The pattern repeats wherever degradation is graceful. A cache that silently serves stale data when the origin is unreachable. A feature that falls back to a default locale. A retry that gives up and returns an empty list. A resolver that returns a last-known-good address. Each is correct engineering, each produces a usable result, and each can run as the only path for months while everything reports healthy.

So the question worth asking of any fallback is how would I know how often this fires — and if the honest answer is that nothing counts it, then the fallback is not a safety net. It is a second implementation, running in production, that nobody is measuring.