It compiles, the test is green, and the screen is a grey box

A screen was specified with colour-coded status chips, a stepped progress header and a drill-down panel. What shipped was a column of grey circles.

Every check in the pipeline passed. The code is well formed; nobody had written a check that looks at the screen.

The case this post is about#

One dashboard tab, shipped and archived as complete, which rendered gray circles where the mockup specified colour-coded chips, a drill-down panel and stepped progress headers.

Every check passed. It compiled, every element the spec named was declared, every handler had a non-empty body, every string was externalised. The indicators that were supposed to be checked:

const INDICATORS: Indicator[] = [
  { id: "state-colours", find: async (p) =>
      (await p.locator('[data-state="error"]').evaluateAll(
        (els) => els.every((e) => getComputedStyle(e).color !== "rgb(128, 128, 128)"))) },
  { id: "drill-down",    find: (p) => p.getByRole("button", { name: /details/i }).isVisible() },

Those two could only have been evaluated against a rendering, and nothing was rendering. This post follows that tab, because the operator found the gap on the first walkthrough and every automated check had already said yes.

The symptom#

The build is green, the composable exists with the right name, the states are declared, the structural scan finds the elements the specification named, and a human opening the app sees the gap in about thirty seconds. That asymmetry — every automated check satisfied, the problem obvious to the first person who looks — is the whole subject.

The second thing is that the implementation is not wrong in any local sense. Somebody produced a composable that takes the right state, branches on it correctly and renders something for every case. It is a faithful skeleton of the design. The design's content — which colour means which state, what the drill-down reveals, how the header steps — is the part that is missing, and none of it is structural.

The third is that the test agrees with the code. A unit test asserts the composable emits a node per item and that the count matches the state. A grey circle is a node. The test is correct, it is testing something real, and it passes identically on the shell and on the finished screen.

The fourth is the one that makes this expensive rather than merely annoying: it is reported as done. The specification said three things, the implementation has three things in it, and the only way to know the three things are not the three things is to look at the pixels. Nothing in the written artifacts disagrees.

There is a fifth that only appears later. Because the screen is reported done, the next piece of work is built on top of it — a navigation entry, a test, another screen that matches its conventions — and by the time anybody notices, fixing it means touching all of that rather than one composable. A defect that is cheap on the day it lands gets more expensive every day it is believed.

Why the obvious fix failed#

Scan the source for the specified elements. The check I built first, and it is satisfied by a shell. The mockup says "status chip"; the source contains a composable named for a status chip, so the scan passes. What it renders is below the resolution of any scan that reads text.

Assert on the semantics tree. Better, and it gets you node presence, roles and labels — real structure, more than a grep. It still does not carry colour, spacing, or whether the drill-down panel actually contains anything, because the semantics tree is a model of accessibility rather than of appearance.

Screenshot tests. The right instrument, and it cannot be the first one. A screenshot test compares against a reference, and the reference is produced from the implementation — so a shell implemented first produces a shell reference, and every subsequent run agrees that the screen looks exactly like the grey boxes it has always looked like.

Have the reviewer check it against the mockup. The thing that eventually caught it, and the reason it is not a solution is throughput. It works when somebody does it, it is skipped under load, and it is precisely the check that does not happen on the twentieth screen of a long week.

It is also the check that is hardest to ask for honestly. "Compare this against the design" sounds like a small request and is actually several minutes of careful looking per screen, against a design file the reviewer has to go and find. Everybody agrees it should happen; the disagreement is about whose afternoon it comes out of.

The actual mechanism#

The gate has to render the artifact and compare the rendering against the design, and the design has to be stated in a form the comparison can read.

designed UX   the specification, in the plan, as explicit indicators
render        the actual artifact — Compose on a device, HTML in a browser
inspect       does the rendering carry the designed indicators
verdict       recorded per task, not per file

The second line is the one that cannot be substituted. Every cheaper check — source scan, semantics assertion, type-level constraint — is answering a question about the program text, and the failure is not in the program text. There is no amount of reading that distinguishes a correct composable from a shell with the same shape, because the difference is in what the shape is filled with.

The sub-case that determines whether this is buildable at all is how the design is stated. A mockup as an image is not something a check can read; a sentence saying "looks right" is not either. What works is a short explicit list of indicators — these states are colour-coded, this panel opens, this header shows progress in steps — written into the plan when the screen is designed. That list is what the rendered check compares against, and writing it costs a minute at the moment somebody already has the design in front of them.

There is a related failure that the same gate catches, and it is more common than the grey box: the screen that was right and has drifted. A token changed, a shared component was refactored, a state was added and renders with the default. Nothing was ever wrong on purpose, and the rendered check is the only thing that notices.

The fix#

state       designed indicators explicitly, in the plan, at design time
render      the delivered artifact, really — browser, device, preview
compare     rendering against the stated indicators
record      a conformance verdict per task, visible, not a log line

The check renders the delivered artifact and compares it against indicators the plan declared, so "designed" is a list of things that must be observable rather than an adjective:

type Indicator = { id: string; find: (page: Page) => Promise<boolean> };

const INDICATORS: Indicator[] = [
  { id: "state-colours", find: async (p) =>
      (await p.locator('[data-state="error"]').evaluateAll(
        (els) => els.every((e) => getComputedStyle(e).color !== "rgb(128, 128, 128)"))) },
  { id: "drill-down",    find: (p) => p.getByRole("button", { name: /details/i }).isVisible() },
  { id: "stepped-header", find: async (p) => (await p.locator("[role=progressbar]").count()) > 0 },
];

export async function conformance(url: string, declared: string[]) {
  const page = await open(url);
  const rows = await Promise.all(declared.map(async (id) => {
    const ind = INDICATORS.find((i) => i.id === id);
    if (!ind) return { id, verdict: "unknown-indicator" as const };
    return { id, verdict: (await ind.find(page)) ? "present" as const : "ABSENT" as const };
  }));
  const bad = rows.filter((r) => r.verdict !== "present");
  // A verdict per task, returned — not logged. A log line is not a gate.
  return { ok: bad.length === 0, rows };
}

On a tab delivered as a structural shell:

$ conformance --screen agentic-state
  state-colours    ABSENT     every indicator renders rgb(128,128,128)
  drill-down       ABSENT     no control matching /details/i
  stepped-header   present
design_conformance: FAIL (2 of 3)

It compiled, it rendered, it passed every test. Three grey circles where the plan said colour-coded chips with a drill-down — which only a rendered comparison can see.

The fourth line matters because of what happens when a gate produces output nobody reads. A verdict attached to the task, surfaced where the work is accepted, is a thing somebody acts on; the same verdict in a build log is a thing somebody finds afterwards while looking for something else.

What makes the whole gate tolerable is that it only has to fire on tasks that have a design. A dependency-injection binding has no appearance and should be able to say so explicitly rather than being forced through a check that cannot apply. The waiver is per task, it names a reason, and its rarity is what keeps the gate honest.

There is a tempting version of this that does not work, and it is worth naming because it is where most people go first: compare the rendering against the mockup image directly. Pixel comparison against a design file fails on everything that legitimately differs — real data instead of placeholder content, the actual type rendering, platform chrome — and produces a diff that is entirely noise with the one real finding somewhere inside it. The explicit indicator list exists precisely because it is the part of the design that should be mechanically true, separated from the part that should merely be close.

How we proved it#

The incident is the proof and it is not flattering. A dashboard tab was specified with colour-coded chips, drill-down and stepped progress headers. It shipped as grey circles, the engine recorded the work as complete because every evaluator passed, and the gap was visible to the first person who walked through the feature. Every check in that pipeline was correct about its own question.

The structural scan's blind spot is demonstrable in one file. Write a composable named for the component the design asked for, render a grey box inside it, and run the scan. It passes. That takes a minute and it settles the argument about whether a text-reading check could ever be sufficient.

What I have not measured is how many drifted screens the rendered gate has caught since. I built it after an incident, I know it blocks the class that caused the incident, and I have not kept a count of subsequent catches — which means I can describe the mechanism honestly and cannot tell you the rate.

What I have not solved: fidelity below the indicator level. The gate catches a grey box where a colour-coded chip was specified. It does not catch a chip that is the right idea and the wrong shade, spacing that is four points out, or a transition that is technically present and feels wrong. Those are real defects, they are what distinguishes a good screen from an adequate one, and the only instrument I have for them is a person.

And a cost I would warn somebody about: rendering is slow and awkward. A browser or a device in the verification loop turns a check measured in milliseconds into one measured in tens of seconds, needs infrastructure that can fail for its own reasons, and produces a category of flake that has nothing to do with the code. Every cheaper check I listed in the failed-fixes section was attractive for exactly this reason, and the reason none of them worked is the same reason this one does.

The thing that changed how I think about it is that this is not a gap in the tooling. A compiler that reasoned about whether a screen matched a design would be a strange and bad compiler. The static checks are complete with respect to the questions they exist to answer, and the question "does this look like what was asked for" was simply never in their domain — so the absence is not a defect to be fixed in them but a check that was never built.

What this generalizes to#

Well-formed and correct are different properties, and most pipelines are built almost entirely out of checks for the first one.

The shape recurs well beyond user interfaces. A schema migration that applies cleanly and produces the wrong shape. A generated API client that compiles against a specification it has misread. A configuration that parses and describes a system nobody wanted. A report that renders and contains the wrong figures. In every case the artifact is valid, every validator agrees, and the only check that would catch it is one that looks at the result rather than at the construction.

So the question worth asking of any pipeline is which of its checks look at the output. If the honest answer is none — if everything inspects the source, the types, the structure and the logs — then the pipeline can tell you the thing was built correctly and nothing at all about whether it is the thing you wanted.