It returned an identifier and produced nothing

A generation call returns. There is a session identifier, a success indication, and no artifact. The thing that was supposed to be produced does not exist.

The accompanying message describes a problem with the request. The actual problem is that the account had run out of quota.

The case this post is about#

One call that returned 200, handed back a session identifier, and created nothing.

No exception, no error field, a plausible id in the response. The artifact it was supposed to produce simply was not there, and the next call in the sequence failed with a message blaming the input — which sent the investigation into the request payload, where nothing was wrong.

The actual cause was upstream and temporary, and the tell was in the shape of the failures: they clustered in time rather than by input. A minimal known-good request failed in the same window, which is the fact that ends the argument.

This post follows that one call, because a success that produces nothing is harder to diagnose than a failure, and an error message naming a cause you control is the most expensive kind of wrong.

The symptom#

A call that neither fails nor works. Failure is easy — there is a code, a message, a path to follow. This returns the shape of success: an identifier that looks like a handle to something, which you can print, log and carry around, attached to nothing.

The second thing is the misdirection, and it is the expensive part. The message mentioned the input. So the first hour went into the input: shortening it, simplifying it, removing the parts that seemed unusual, bisecting it to find the offending clause. All of that is competent debugging of a hypothesis the error text supplied, and the hypothesis was wrong.

The third is that the input genuinely does affect it, which made the false trail convincing. A shorter request sometimes succeeded, because a smaller unit of work occasionally fit inside whatever remained. So the evidence appeared to confirm the theory — simplify and it works, elaborate and it fails — and that correlation is real and causally backwards.

Backwards correlations of this shape are the most durable kind of wrong belief, because every test confirms them. The smaller request succeeding is not evidence about size; it is evidence that a smaller unit of work has a better chance of fitting into whatever capacity remains — which produces the same observations as a size limit and implies a completely different fix.

The fourth is the timing pattern that eventually gave it away. Failures came in runs. Several minutes of everything failing, then everything working, with no relationship to what was being asked for. Input-dependent failures do not do that; they fail on the same input every time, which is the observation that finally separated the two theories.

It is worth noticing how long that took, because the information was available from the first hour. Every failure had a timestamp and every timestamp was in the terminal scrollback. The pattern was sitting in data I already had, unexamined, because I was looking at what I sent rather than at when I sent it.

Why the obvious fix failed#

Read the error message. Normally the right first move and here it is the trap. The message was produced by a layer that knew the call had not succeeded and did not know why, so it supplied the most common reason — which is a reasonable default and was not what happened.

Simplify the request until it works. Produces a working request and no understanding. It also reinforces the wrong model, because it does work, and you come away believing you have found a limit in the input rather than a limit in the account.

Retry on failure. Correct behaviour that would have masked the diagnosis entirely. Retries would have made the symptom intermittent-and-mostly-fine, which is the state in which nobody investigates anything.

That is the genuinely awkward part of this story, because retrying is correct. The behaviour that makes a system robust is the same behaviour that converts a diagnosable outage into a background hum, and both properties come from the same line of code. The resolution is not to stop retrying but to count the retries somewhere a person will see the number.

Check the response status. The status said success. Every field a caller would normally interrogate was present and well-formed. There was nothing to check that would have reported the truth, which is the actual defect in this interaction.

The actual mechanism#

Treat a response that contains no artifact as a failure, regardless of what it says about itself, and probe the hypothesis the message is pointing you away from.

assert    the artifact exists, not just that the call returned
probe     a minimal known-good request — if that fails too, it is not your input
observe   whether failures cluster in time or cluster by input
believe   the clustering over the message

The second line is the single most useful thing here, and it took me far too long to run. A one-line request that has worked a hundred times is a control: if it fails, the variable is not the input. That is a thirty-second experiment that would have replaced an hour of bisection, and the reason I did not reach for it is that the message had already told me where to look.

The sub-case worth stating is the shape of the clustering, because it distinguishes the two theories cleanly. Input-dependent failures are reproducible on demand and stable across time. Capacity-dependent failures are reproducible across inputs and unstable across time. One look at when the failures happened, rather than at what was sent, separates them — and nothing about the response encourages you to look at timing at all.

There is a related trap on the other side, which I also fell into. Having concluded it was capacity, I over-generalised and started attributing every subsequent failure to it, including one that really was a malformed request. A theory that explains the last outage is not a theory that explains the next one, and the same control probe settles it in both directions.

Both errors have the same root, which is answering a diagnostic question with a stored conclusion rather than a fresh observation. The first time the stored conclusion came from the error message; the second time it came from me. A control probe is cheap specifically because it does not care which of those is doing the misleading.

The fix#

verify     the artifact, as the success condition — never the status alone
control    keep a minimal known-good request and run it when things fail
plot       failures against time before plotting them against input
distrust   an error message that names a cause the caller can control

The control request is the instrument, and it is what separates "my input is wrong" from "the service is degraded" — two diagnoses that produce completely different afternoons:

const CONTROL = { prompt: "hello", size: "256x256" } as const;   // minimal, known-good

export async function callWithControl<T>(req: Request, verify: (r: Response) => T) {
  const res = await api.call(req);

  // SUCCESS = the artifact is there, not that the call returned.
  try {
    return { ok: true as const, value: verify(res) };
  } catch (artifactErr) {
    // The response was accepted and produced nothing usable. Before believing the
    // error message — which will blame the input — ask the control.
    const control = await api.call(CONTROL).then((r) => {
      try { verify(r); return "ok" as const; } catch { return "also-empty" as const; }
    }).catch(() => "unreachable" as const);

    throw new Error(
      `call returned ${res.status} and produced no usable artifact.\n` +
      `  control request: ${control}\n` +
      (control === "ok"
        ? "  the service is fine — the fault is in this request"
        : "  the control also failed — this is the SERVICE, not your input"),
    );
  }
}

The same failure, diagnosed two ways:

$ generate --asset cone-anim
error: call returned 200 and produced no usable artifact.
  control request: also-empty
  the control also failed — this is the SERVICE, not your input

The error body had said the prompt was too complex. I spent an hour simplifying prompts before running the control, which answered it in nine seconds.

The fourth line generalises the whole episode. An error produced by a system that cannot see the real cause will describe something it can see, and what it can see is your request.

Which means the useful reading order is roughly the inverse of the natural one: start with what the message cannot possibly know, because that is where the unexamined ground is, and treat the confident specific part as the hypothesis most likely to have been supplied by a component guessing. The messages most likely to mislead are therefore precisely the ones that sound most actionable, because actionability is what the layer writing them was reaching for.

What makes the control probe worth institutionalising is that it costs nothing until you need it, and when you need it you are in exactly the state where you will not think of it. The right time to write it down is while things are working.

Mine is a single short request with a known-good input, kept beside the integration rather than in a notebook somewhere, so running it is one command and not an act of reconstruction. The value is entirely in it being reachable without thinking, because the moment it is needed is the moment nobody is thinking clearly.

How we proved it#

The control probe separated the theories in thirty seconds. A trivial request, known good, run during a failure window, failed. That single observation eliminates every input-related explanation at once — and it is available at any moment to anybody who thinks to run it.

The temporal clustering is visible once you look for it. Failures arrived in sustained runs rather than scattered, and recovered wholesale rather than per-request. That pattern does not occur for input-dependent faults, and plotting failures against time rather than against content is what made it obvious.

What I cannot tell you is the real quota state. I inferred exhaustion from behaviour; I never saw a counter, because none is exposed. The inference fits everything observed and I would not present it as confirmed — the honest claim is that the failure was not input-dependent and behaved exactly as a capacity limit would.

What I have not solved: distinguishing a capacity limit from a transient service fault. Both cluster in time, both recover wholesale, both are invisible to the caller. The practical response is identical — back off and retry later — so the ambiguity has not cost me anything yet, and it would matter immediately if the two needed different handling.

And a limitation of the artifact-check approach: it catches the empty response and says nothing about a response containing the wrong artifact. Verifying existence is a much weaker check than verifying correctness, and everything above would pass happily on an output that was produced successfully and is not what was asked for.

That is a different and harder problem, and worth keeping separate rather than letting the two blur. Existence is binary, cheap and mechanical. Correctness requires knowing what was wanted, which for generated artifacts usually means somebody looking. Conflating them produces a check that feels thorough and verifies the easy half.

The thing that reframed it was noticing how much authority I had given a sentence written by something that did not know the answer. The message was generated by a layer handling an error it could not characterise, which is the condition under which messages are least reliable — and I had treated it as the most authoritative evidence available because it was the most specific.

What this generalizes to#

The most specific-sounding explanation is often produced by the component least able to know, and specificity reads as credibility.

This is the shape of a great deal of wasted debugging. A client library reporting a parse failure on a response that was actually a truncated gateway error. A database driver reporting a syntax problem on a query rejected for permissions. A build tool naming a missing file when the real issue is a path it never searched. In each case something near the surface describes what it can see, and what it can see is the input you supplied.

So the useful discipline when a message points at something you control is to check the thing it is pointing away from first — because the layer that wrote the message did not have access to that, which is exactly why it did not mention it.

The corollary is a reasonable default for reading any error: ask what the component writing it could actually observe. A message from a client library can describe the request and the bytes that came back, and nothing about the state of the service. Anything it says beyond those two is inference, offered in the same confident tone as the parts it knows.