The third time this filled up, I diagnosed it as an outage

TL;DR#

A generation service started returning success with an empty body. Every call. I treated it as an outage, retried about thirty-six times across several hours, and wrote guidance telling the next person to retry patiently.

It was not an outage. The project I was generating into had filled up — a consumable with a capacity of a few hundred items. It had filled up twice before, in the previous two weeks. Both times someone replaced it by hand and moved on, and neither replacement was written down.

So the third occurrence of a routine, predictable end-of-life arrived looking exactly like a service failure, because nothing in the system recorded that the resource had a lifetime at all.

The case this post is about#

A hosted design service. You create a project, then generate assets into it. Our pipeline had one, recorded in a state file, shared by every asset so they inherit a common design system.

Then every call started coming back like this:

response keys: projectId, sessionId
outputComponents: []

HTTP 200. No error. No message. Just nothing where the generated asset should be.

Connection fine, authentication fine, project resolution fine. A one-line prompt — "a single teal square on a white background" — behaved identically to a 300-word art brief. The service was accepting the work and returning an empty envelope.

The symptom#

What makes this expensive is that an empty success looks exactly like transient trouble, and every instinct points at retrying.

attempt 1: empty    attempt 2: empty    attempt 3: empty
… 25s backoff …     … 120s backoff …
attempt 15: empty   attempt 22: empty   attempt 36: empty

Once, somewhere in the middle, a call succeeded. One. That single success did more damage than all the failures, because it made every wrong theory survivable: the service is flaky, the rate is low, keep going, you will get through.

I built a theory on it — the successful call had used a short prompt with no flags, so the long brief must be the problem — and then disproved my own theory with a control, which was the first honest thing I did. But I replaced it with a worse one: low-yield throttling, retry patiently. I wrote that into the codebase as guidance. One coin-flip, read as a base rate.

Why the obvious fix failed#

"Retry with backoff." Thirty-six times, up to two minutes apart. A full capacity ceiling does not clear with time, so the retry loop had no terminating condition that could ever be satisfied.

"Shorten the brief." The prompt was irrelevant — a six-word probe failed identically. The codebase even said so in large letters, and I half-believed it only because the one success happened to be short.

"Use a different account." Reasonable, and I refused it for a reason that turned out to be right for the wrong reason: the design system is bound to the project, so switching accounts produces assets that do not match the existing set. I declined on aesthetics while the actual problem was one layer down.

"It is the service." The most plausible reading and the one I committed to. Nothing available from inside a single project distinguishes this service is down from this container is full. Both present as a clean, empty success.

The actual mechanism#

The resource had a finite capacity, and nothing in the system said so.

When I finally looked at where our existing assets had actually been generated, the history was sitting in the provenance sidecars the whole time:

project 18393347132512223580   226 assets   Sep 27 → Oct 1 11:00    replaced
project  2604303785906022147    67 assets   Oct 1 12:15 → 18:05     replaced
project  6284002269741766662   373 assets   Oct 1 18:16 → Oct 7     ← the "outage"

Three projects. Two prior end-of-life events, in a fortnight. Each one noticed by a human, fixed by hand in minutes, and never recorded as anything.

Three properties turned a routine lifecycle into an incident:

The failure mode is a success. No quota error, no 429, no "capacity reached". An empty 200 is the least informative thing an API can return, and it is what you get.

The replacements left no trace. Someone minted a new project twice and the pipeline carried on. The only evidence was a project id changing inside per-asset metadata — technically present, semantically invisible, and nobody reads provenance until something breaks.

The distinction that matters — exhausted container versus broken service — is not visible from any number of retries against the same container. It needs a control, and the control has to be a fresh one:

type Verdict = "container-exhausted" | "service-down" | "healthy" | "inconclusive";

async function diagnose(existingId: string, mint: () => Promise<string>): Promise<Verdict> {
  const probe = async (id: string) => {
    let ok = 0;
    for (let i = 0; i < 4; i++) {                 // 4 identical calls, same prompt
      const r = await generate(id, PROBE_PROMPT);
      if (r.outputComponents?.length) ok++;
    }
    return ok;
  };

  const incumbent = await probe(existingId);
  if (incumbent === 4) return "healthy";          // nothing to diagnose

  // The load-bearing step. Retrying `existingId` harder cannot distinguish the two
  // causes, because BOTH return an empty 200 forever — which is exactly why a week
  // was spent retrying one of them.
  const control = await probe(await mint());

  if (control === 4 && incumbent === 0) return "container-exhausted";
  if (control === 0 && incumbent === 0) return "service-down";
  return "inconclusive";                           // partial — do not guess, escalate
}

Run against the project that had been declared an outage:

$ deno run -A diagnose.ts --project 6284002269741766662

incumbent 6284002269741766662 : 0 / 4   (empty 200, no outputComponents)
control   10158506550203694103: 4 / 4
verdict   container-exhausted

Ninety seconds, and it ends an argument that had run for days. Note what makes it work: the control is minted during the diagnosis rather than read from config. A spare project created last week and left idle would be a worse control, because "fresh" is the variable under test.

The guidance actively pointed away from it. Someone diagnosed this correctly on Oct 1, with the right experiment: same prompt, old project empty, fresh project works first try. Later, someone else wrote in the same comment that this had been coincidence and that creating projects "only creates sprawl."

Those two claims sat four lines apart, in identical prose, with nothing marking that the first had a control behind it and the second was an impression. I read both and believed the second — then wrote a third, worse one underneath. The guess outranked the measurement because they were formatted the same.

The fix#

The experiment takes ninety seconds. One variable, several trials each:

// Same prompt. Same account. Same connection. Same minute.
// The ONLY difference is which project the call names.
for (const i of trials) record(generate(EXISTING_PROJECT, "a single teal square"))
const fresh = await createProject({ title: "ab-probe" })   // control arm
for (const i of trials) record(generate(fresh.id,        "a single teal square"))

RESULT   existing = 0/4      fresh = 4/4
re-run   existing = 0/3      fresh = 3/3

Decisive, reproducible, and it settles in two minutes a question that thirty-six retries could not touch — because retrying only ever samples one arm.

Two details decided whether the probe was evidence or noise:

Three outcomes, not two. My first version returned a boolean and swallowed exceptions as false. It reported 0/4 for the existing project — which looked like a result and was actually my own client code throwing. A probe that cannot distinguish the service returned nothing from my code broke manufactures the exact false measurement it exists to prevent.

Read the schema, do not guess it. I guessed the creation call took name; it takes title. The resulting "invalid argument" read like a service fault and was mine. The live schema was one introspection call away.

The durable fix is not the probe, though. It is recording the lifecycle: when a consumable is replaced, write down what replaced it, when, and how much it had produced. Three rotations happened; a one-line record on the first would have made the third a known maintenance event instead of an incident.

How we proved it#

The A/B result above is the proof of the mechanism, and reproducing it on a second independently created project is what rules out "the first fresh project was lucky".

The history needed separate evidence, because "it filled up before" is a claim about the past. That came from counting provenance sidecars by project id — 226 / 67 / 373, with first and last timestamps — which is a record that existed all along and that nobody had ever aggregated.

One thing I still cannot explain, and will not paper over: asking the service to list the project's screens returns zero, despite 373 recorded generations against it. Either the listing is scoped differently from what I assume, or screens are cleared after capture. It does not weaken the diagnosis — the control settles that on its own — but it does mean I cannot yet use a count as an early warning, which is the thing I actually want.

What this generalizes to#

An empty success is a failure mode, and it is the worst-shaped one. Anything that returns 200 with nothing in it will be read as transient. If you operate something that can fill up, make it say so; if you consume something that can, assume the empty envelope means full until you have ruled it out.

Consumables need recorded lifetimes. Anything with a capacity — a project, a key, a quota, a licence seat, a disk — will reach end-of-life, and the replacement is cheap. What is expensive is the second and third time, when the knowledge that it is routine lives only in someone's memory of having done it before.

Write down the replacement, not just the fix. The two previous rotations worked perfectly. Nothing was broken by them. The damage was done by their silence, and a single line — rotated project X→Y at 226 assets — would have turned my outage into a five-minute maintenance task.

Mark which claims have evidence. Two contradictory sentences sat in one comment, one from a controlled experiment and one from an impression, visually identical. When you overturn a measured claim, re-run the measurement or say explicitly that you did not. Prose does not carry provenance unless you put it there.

A control beats a retry, always. Retrying samples one arm harder. It can tell you something is not working; it can never tell you why, because the thing you need to vary is the thing you are holding fixed.

Related#

• 163 checks reported green. None of them read the published page. — the other empty-success failure
• 429 is not an error, it is a schedule — when retrying IS the right answer
• A stale doc does more damage than a missing one — confidently wrong, by a different route

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.