Reading back is not optional

An agent generates an image, writes it to disk, and reports that the image was generated. Every part of that sentence is true.

The file is eight hundred bytes of error page with a .jpg extension.

The case this post is about#

The example I will carry through is the one that cost me most recently, and it is small enough to reproduce in a terminal. A generator script writes an illustration for a blog post, and a second command converts it:

deno run -A scripts/stitch-asset-generate.ts \
  --asset cone-anim --kind anim --intent-file brief.txt --duration "9s linear"
deno task anim-to-gif stitch/cone-anim.html posts/<slug>/assets/anim.gif

The second command prints a JSON summary ending "moving": true, which is as close to a success signal as that pipeline has. On three separate occasions it was true and the artifact was wrong: a GIF of 36 identical frames, a GIF that was byte-for-byte the previous run's output, and a header rendered portrait when the post needed landscape. Nothing failed, nothing logged, and "moving": true was not a lie — it was an answer to a question I was not asking.

The symptom#

The report describes an intention rather than an outcome. The call was made, it returned without raising, a file exists at the path — and the agent has treated the absence of an error as evidence about the content, which it is not.

The second thing is that the failure is usually not a failure anywhere. The service accepted the request, returned something, and wrote it down. There is no exception to catch, no status code to check, nothing in the mechanical record that distinguishes this from the successful case. The only place the difference exists is in the artifact.

The third is how convincing the report is. "Generated the illustration and saved it to assets" is specific, confident, and exactly what a successful run would say. Nothing in the language of the claim encodes whether anybody looked.

The fourth is the compounding. An unverified artifact becomes an input to the next step — wired into a page, referenced from a manifest, counted in a total — and each subsequent step treats it as real, so the error surfaces far from where it was made, in a form that no longer looks like a generation problem.

This is the property that makes it expensive rather than annoying. By the time a broken image is noticed on a published page, the run that produced it is long finished, and reconstructing what happened means going back through several layers of work that all reported success.

The fifth property is the one that makes it recur: skipping the read-back is not laziness, it is a reasonable inference. The call returned, the file is there, nothing threw — a person reasoning carefully from the available evidence would conclude the same thing. The evidence is simply about a different question than the one being answered.

Why the obvious fix failed#

Check the exit code. The baseline, and it is answering a different question. An exit code tells you the process finished; it says nothing about what the process wrote, and for anything that goes through a network it is routinely zero on a response that is an error document.

Have a second agent check the first one's work. The expensive answer, and it inherits the same blind spot unless somebody specifies what to check. A reviewer told to confirm the artifact was produced will confirm the artifact was produced.

Check the file exists. Cheap, and it passes on the error page. Existence was never in doubt — something was written, and the question was what.

Check the file size. Better, and it is a heuristic with a threshold somebody has to pick. It catches the empty file and misses the plausible-sized wrong one, and the threshold becomes a thing that needs maintaining per artifact type.

Trust the service's own success field. Reasonable when it exists and it reports on the request, not on the result. A generation service can quite correctly report that it generated something, and the something can be unusable for the purpose it was requested for.

The actual mechanism#

Read the artifact back, in the form it will be consumed in, and look at it.

produce   make the artifact
read      load it back from where it was written
inspect   in its real form — view the image, parse the file, run the output
assert    the specific property the work was supposed to produce

The third line is where almost all the value is, and it is the one that gets substituted away. Reading bytes and checking a magic number is not inspecting an image; it establishes that the file is an image and says nothing about whether it is the image that was asked for. An agent that can see pictures should look at the picture.

The fourth line is what separates this from a smoke test. "The file is a valid PNG" is a property of the format; "this illustration shows three objects where two of them disagree" is a property of the work. Only the second one is evidence that the step did its job.

The sub-case that is easy to miss is the artifact that is valid and stale. A generation call that silently does nothing leaves the previous run's output in place, and every format check passes because the file is a perfectly good file from an hour ago. The only reliable detection is comparing against what was there before — content, not timestamp.

A second sub-case, specific to anything with frames or sequence: a valid animation whose frames are all identical. It plays, it is the right length, it reports as moving, and nothing is moving. That one is only caught by comparing frames to each other, which nobody does unless they have been burned.

Both sub-cases share a shape worth naming: the artifact is fine and the run did nothing. Format checks interrogate the artifact, so they are structurally incapable of catching this class, however many of them you add. The only thing that sees it is a comparison across time or across frames.

The fix#

read-back     every produced artifact is loaded again from its destination
real-form     images viewed, data parsed, code run — not inspected as bytes
differential  compare against the previous artifact; identical is a failure signal
specific      assert the property this step existed to produce

The third line is the one I would add first if I could only have one. Most silent-no-op failures are caught by it, it costs a hash, and it needs no judgement about thresholds — a generation step that produced byte-identical output to the last run did not run.

The fourth is the hardest to automate and the most valuable, because it is the only check that notices the artifact is wrong rather than absent. In practice it often means an agent describing what it sees and comparing that against what it meant to make, which is slower than a format check and is the actual verification.

What makes all of this tractable is that it is cheap relative to production. Generating an illustration is expensive; looking at it is not. The reason the step gets skipped is not cost — it is that the work felt finished when the call returned.

On the running example the whole read-back is four lines, and each maps to a line of the contract above:

# real-form — look at the thing, not at its bytes
magick posts/<slug>/assets/anim.gif -coalesce /tmp/fr/f%03d.png

# specific — the property this step existed to produce: the frames differ
ls /tmp/fr | wc -l                                           # 108 frames
shasum /tmp/fr/*.png | awk '{print $1}' | sort -u | wc -l     # 106 distinct → moving

# differential — identical output to the last run means the generator no-opped
shasum posts/<slug>/assets/anim.gif

The 36-identical-frames case fails on the second check: 108 against 1. The silently-reused output fails on the third, where the hash matches the previous run exactly. Neither is caught by "moving": true, and both are caught by about eight seconds of shell.

As a function the generator calls before it is allowed to report anything, it is short enough to have no excuse for being absent:

// verify-artifact.ts — runs INSIDE the producing step, not in a later pass.
import { crypto } from "@std/crypto";

const sha = async (p: string) =>
  [...new Uint8Array(await crypto.subtle.digest("SHA-256", await Deno.readFile(p)))]
    .map((b) => b.toString(16).padStart(2, "0")).join("");

export async function verifyAnimation(out: string, previous?: string) {
  const st = await Deno.stat(out).catch(() => null);
  if (!st || st.size === 0) throw new Error(`no artifact at ${out}`);

  // differential — a generator that produced byte-identical output did not run
  const digest = await sha(out);
  if (previous && digest === previous) {
    throw new Error(`${out} is byte-identical to the previous run — the step no-opped`);
  }

  // specific — the property this step existed to produce: the frames differ
  const dir = await Deno.makeTempDir();
  await new Deno.Command("magick", { args: [out, "-coalesce", `${dir}/f%03d.png`] }).output();
  const frames: string[] = [];
  for await (const e of Deno.readDir(dir)) frames.push(await sha(`${dir}/${e.name}`));
  const distinct = new Set(frames).size;
  if (frames.length > 1 && distinct === 1) {
    throw new Error(`${out} has ${frames.length} identical frames — nothing is moving`);
  }
  return { digest, frames: frames.length, distinct };
}

Run on a GIF whose frames are all the same — the case that reports "moving": true:

$ deno run -A verify-artifact.ts posts/x/assets/anim.gif
error: posts/x/assets/anim.gif has 36 identical frames — nothing is moving

$ deno run -A verify-artifact.ts posts/x/assets/anim.gif    # after a real regeneration
{ digest: "9f2c…", frames: 108, distinct: 106 }

Which is why the read-back belongs inside the step rather than in a later verification pass. A separate checking stage runs after the agent has already reported, so the report is wrong for as long as it takes somebody to read the check — and in an autonomous drive, the next step has consumed the artifact by then.

How we proved it#

Every instance I have hit was found by reading back, and none by any other check. The error page with an image extension, the animation whose frames were all the same, the generation call that silently throttled and left the previous file in place: each passed existence, size and format checks, and each was obvious the moment the artifact was actually looked at.

The identical-output check is the cheapest real win. It caught a throttled generation that had reported nothing wrong — the script printed no success line, I converted the file anyway, and the hash matched the previous run exactly. Nothing else in the chain would have noticed.

What I have not solved is subtle wrongness. An image that is a plausible illustration of something slightly different passes every mechanical check and most inspections, because it looks right at a glance. Catching that requires knowing what was asked for in enough detail to compare, and that comparison is a judgement I have not found a way to make reliable.

What I did find workable is asking for the specific thing rather than for an opinion. "Does this show three objects, one of which differs from the other two" is answerable and checkable; "is this a good illustration" is not, and anything asked the second question will say yes.

And a limitation worth naming: read-back verifies the artifact at the moment it was produced. It says nothing about whether it survives the next step — an image that is correct on disk can still be referenced by the wrong path, resized badly, or silently excluded from a build. Each of those needs its own read-back at its own boundary.

The thing that reframed it was noticing that "I generated the file" and "the file is what I said it is" are different claims and only one of them is usually tested. The gap between them is exactly where confident wrong reports come from, and it is not a gap that more careful wording would close.

It also explains why this failure is so much more common for agents than for people. A person who makes an image sees it as a side effect of making it; an agent does not see anything unless it decides to look, so the verification that is free for a human is an explicit step that has to be remembered.

That asymmetry suggests where to put the effort. Rather than trying to remember, make the production step structurally incapable of finishing without a read-back — the tool that generates returns the artifact's content, not a path, so there is nothing to report until something has been looked at.

What this generalizes to#

Confirmation that an operation was performed is not evidence about what it produced.

The shape is everywhere once you look for it: a deployment that reported success and served the old build, a migration that ran and changed nothing, a cache write that silently dropped the value, an email that was accepted by the relay and never delivered. In every case the mechanism reports on itself and the thing it was supposed to affect is unexamined.

So the question to ask of any step that claims success is what did it look at. If the answer is "the response", the claim is about the request. The artifact is a separate thing and it will cheerfully be wrong without telling anybody, because being wrong is not an error.