A verification step that can pass while the bug is present is not a verification step

Four artifacts shipped past every automated check available in this project during one session, and every one of them was wrong.

Not wrong in a way the checks were too weak to catch. Wrong in a way no check of that kind can catch, because the checks inspect what was written and the defect is in what was drawn.

TL;DR#

• A green build and a screen that renders wrong are indistinguishable to CI — compile-truth is not device-truth.
• The defect class only shows up on a real device, which is exactly where most pipelines stop looking.
• What to assert on so that 'it works' means something falsifiable.

The case this post is about#

One dashboard tab, and four claims made about it:

compiles       pass
declared       pass
handler-wired  pass
looks-right    undetermined  claim "looks-right" needs rendered evidence; got ast

Three passes, and that is exactly what shipped: a tab that compiled, declared every element the spec named, wired every handler — and rendered gray circles where the mockup specified colour-coded chips, a drill-down panel and stepped progress headers.

The fourth row did not exist at the time. The appearance claim was being settled by the same AST evidence as the row above it, which is a claim about source answering a question about output.

This post follows those four rows, because the operator found the gap on the first walkthrough and every check had already said yes.

The symptom#

A generated screen compiles. Its structure validates. A parser confirms every required element is present. A grep confirms the handler is not empty. Everything passes.

Then you look at it and the layout is wrong — an element off-screen, a state that never renders, a colour-coded indicator that came back as a grey circle.

The gap is not laxity. It is category: static analysis operates on the source, and rendering is a function of the source and a layout engine and a device and a set of resources. A checker reading the first term cannot evaluate the product.

This has a specific shape worth naming, because it is not the same as "there was no test":

• a test that exercises a component which renders incorrectly still passes
• a structural check that requires an element finds it in the source, wherever it ends up on screen
• an assertion on a handler's existence says nothing about what the handler does visually

Each is doing its job. The job does not include looking.

Why the obvious fix failed#

The obvious fix is more static checks. Verify more attributes, assert more structure, tighten the parser.

It moves along an axis that does not reach the defect. A layout can satisfy every structural assertion you can express about its source and still draw something nobody would ship, because the thing that went wrong happened after the source was read.

The second attempt is to check the output artifact — but statically. Parse the generated markup, confirm the expected nodes exist. This feels like progress and is the same category error one level down: a node existing in a document is not the same as it being visible, positioned, or legible.

The third, which works and is expensive, is a human looking. That does not scale to a hundred artifacts, which is how the four in question shipped.

The actual mechanism#

Require a check whose input is the rendered result, produced by the real engine.

For a mobile screen that means an on-device capture, not an inspection of the source tree. For an image it means loading the pixels. For an animation it means extracting the frames and comparing them.

The property that makes this work is that the check consumes the same artifact a user would. Every transformation between source and screen has already happened by the time the check runs, so none of them can hide a defect from it.

The cost is real: rendering is slower than parsing, needs a device or a browser, and produces an artifact you then have to evaluate. That cost is the reason the static checks exist and the reason they are not sufficient.

The fix#

What I actually changed was to make the rendered check mandatory for a specific claim rather than generally encouraged.

The claim is narrow: this thing looks right. Any assertion of that form requires evidence produced by rendering — a capture, extracted frames, loaded pixels. No amount of structural validation substitutes, and a run that cannot render reports a third state rather than a pass.

Everything else keeps using the cheap checks, which are excellent at what they cover: does it compile, is the element declared, is the handler non-empty, is the string externalised. Those catch a large class of defect quickly and should run first.

The split is not static-versus-dynamic as a preference. It is: match the check's input to the claim being made. A claim about source is checkable from source. A claim about appearance is not.

Written as a gate, the claim is the input and the evidence is what decides the verdict — which is what makes the third state unavoidable rather than optional:

type Claim = "compiles" | "declared" | "handler-wired" | "looks-right";
type Evidence =
  | { kind: "source"; files: string[] }
  | { kind: "ast"; nodes: number }
  | { kind: "render"; capturePath: string; pixelsRead: number; afterForceStop: boolean };

type Verdict = "pass" | "fail" | "undetermined";

const NEEDS_RENDER: Claim[] = ["looks-right"];

function adjudicate(claim: Claim, ev: Evidence | null, reason: string[]): Verdict {
  if (!NEEDS_RENDER.includes(claim)) {
    return ev ? "pass" : "fail";          // source claims, source evidence
  }
  // An appearance claim with no rendered evidence is NOT a pass, and it is not a
  // failure either — nothing was observed. Collapsing it either way is the bug.
  if (!ev || ev.kind !== "render") {
    reason.push(`claim "${claim}" needs rendered evidence; got ${ev?.kind ?? "none"}`);
    return "undetermined";
  }
  if (ev.pixelsRead === 0) {
    reason.push(`capture ${ev.capturePath} exists but no pixels were read back`);
    return "undetermined";
  }
  if (!ev.afterForceStop) {
    // A capture taken without stopping the process first may show the OLD
    // composition — the install does not kill a running app.
    reason.push(`capture predates a force-stop; may show the previous build`);
    return "undetermined";
  }
  return "pass";
}

The dashboard tab that shipped as a gray stub, run through all four claims:

const checks: Array<[Claim, Evidence | null]> = [
  ["compiles",      { kind: "source", files: ["AgenticStateTab.kt"] }],
  ["declared",      { kind: "ast", nodes: 34 }],
  ["handler-wired", { kind: "ast", nodes: 6 }],
  ["looks-right",   { kind: "ast", nodes: 34 }],
];
for (const [claim, ev] of checks) {
  const why: string[] = [];
  console.log(`${claim.padEnd(14)} ${adjudicate(claim, ev, why).padEnd(13)} ${why.join("; ")}`);
}

compiles       pass
declared       pass
handler-wired  pass
looks-right    undetermined  claim "looks-right" needs rendered evidence; got ast

Three passes and that is exactly what shipped — a tab that compiled, declared every element the spec named, wired every handler, and rendered gray circles where the mockup specified colour-coded chips and a stepped progress header. The fourth line is the only one that was ever going to catch it, and before this change it did not exist: the appearance claim was being settled by the same AST evidence as the row above it.

How we proved it#

Four artifacts from this session, each of which passed everything automated and failed on inspection:

animation      reported moving: true, frames 41 of 43 identical
animation      rendered 2 of 3 briefed beats, no error
cover image    subject placed where a scrim would erase it
hero composite headline collided with artwork, 25 chars inside the limit

The first is the cleanest example. A converter reports whether a GIF moves, derived from compressed size against a single frame. It returned true. Extracting the frames and hashing them showed 41 of 43 identical — the file was technically not static and visually was.

Nothing in that pipeline was broken. The converter's heuristic was reasonable, the field was correctly typed, the value was sincerely derived. The verification was simply answering a cheaper question than the one that mattered.

The fourth is the one that most resembles a normal engineering bug. I had written a rule capping headline lines at roughly 26 characters to avoid collision with artwork. A line of 25 characters collided anyway, because the real constraint is rendered width and the rule measures count. A character-count check passes; a look at the composite does not.

The check I still cannot write#

Being honest about the ceiling: rendering-based verification tells you what appeared. It does not tell you whether what appeared is correct.

I can prove an animation has distinct frames. I cannot mechanically prove those frames show the idea the brief asked for — that took a person looking at a still and saying "the traveller is missing". I can prove a layout drew. I cannot prove it drew the designed thing.

So the rendered check closes one gap completely and leaves the next one open. What it buys is that the remaining judgement is made against the real artifact rather than against a description of it, which is a much better position even though it is still a judgement.

The practical shape of that: every generated image in this session was read back and looked at, every animation had its frames extracted and counted. That is a real per-artifact cost, and it is the only reason those four defects were found at all.

Why this gets worse with generated work#

The category error is old. What changed is the volume.

A person writing a screen by hand looks at it while writing — the render check happens continuously, informally, as a side effect of working. Nobody designed that check and it catches an enormous amount, which is why it was safe to leave it undesigned.

Generated work removes it. A hundred artifacts arrive having been described correctly, and the informal look that used to happen a hundred times now happens once, at review, against a diff. The check did not get weaker; the thing it depended on quietly stopped occurring.

That is the argument for making the rendered check explicit rather than assumed. It was always the load-bearing one, and it used to be free.

What this generalizes to#

For any check you rely on, ask what its input is, then ask what your claim is about. Where those differ, the check can pass while the claim is false — and it will, eventually, in a way that looks exactly like success.

Keep the cheap static checks; they are fast and they catch a lot. But for any claim about how something looks or behaves, require evidence produced by the thing that actually renders or runs it. And leave room for the answer to be could not determine, because a check that must return pass or fail will return one of them about a case it never evaluated.

---

Related#

• A canary that cannot fail proves nothing — When the check itself is structurally incapable of failing
• Ask a second agent to refute the first — Getting a second opinion that is actually adversarial

Keep reading#

One post a week on shipping software with AI agents — what broke, what the fix was, and the evidence. No digests, no course pitch. Subscribe.