How do you test a screen whose text is generated?

A screen that renders model output has no golden value. Run it twice and the text differs; both runs are correct.

Every test you would normally write asserts equality against something. So the question is what to assert instead — and the failure mode is not that you cannot find anything, it is that you find something too easily.

The case this post is about#

One generated animation, and the assertion that cannot be written:

$ assert-varies anim-a.gif
error: anim-a.gif: 36 frames, 1 distinct — it plays and nothing moves
$ assert-varies anim-b.gif
{ frames: 108, distinct: 106, digest: "9f2c1b05…" }

There is no expected output. The generator produces something different every run, so no reference image exists and no string comparison is possible — which is usually where testing stops and eyeballing starts.

The first line is what eyeballing missed: a 36-frame animation where every frame is identical. It plays, it has a duration, it has the right dimensions, and nothing in it moves.

This post follows those two commands, because almost everything around the unassertable value is assertable, and the hard part is keeping the assertion tight enough to still fail.

The symptom#

The instinct is a snapshot test, and it works exactly once. The second run produces different text, the comparison fails, and the fix everyone reaches for is to regenerate the snapshot — which turns the test into a recording of whatever happened most recently.

Loosen it and you slide toward the opposite failure. Assert the text is non-empty and you have a test that passes on the string "x". Assert it contains a keyword and you have a test that passes on a sentence saying the opposite. Both are green, both tell you nothing, and neither will ever go red for a reason you care about.

The gap between those two failures is where the real work is. A test on nondeterministic output is useful only in a narrow band: tight enough to fail on a real defect, loose enough to survive a correct run.

Why the obvious fix failed#

The obvious fix is to make generation deterministic for tests — fix the seed, stub the model, return a canned response.

That is right for the surrounding logic and wrong for the screen. A stubbed model tests that your rendering works when handed a known string, which is worth having and is not the question. The question is whether the screen holds up against the range of things the model actually produces, and a canned response has removed exactly that.

The second attempt is to assert on properties of the string: length bounds, character classes, no placeholder markers. Better, and still string-shaped — it checks the payload rather than the screen, so a correct string rendered into a broken layout passes.

The actual mechanism#

Assert the invariants of the surface, not the content in it.

The useful ones are structural and they hold regardless of what the model said:

• the element that should receive the output exists and is reachable by a stable identifier
• it is non-empty and within the bounds the layout was designed for
• required surrounding structure is present — the label, the action, the state indicator
• forbidden content is absent: placeholder markers, error sentinels, the loading state

The last one carries more weight than it looks. Absence assertions are the ones that catch the failures you did not imagine — a screen stuck in loading, a fallback string that shipped, an empty state rendered over real content. They are cheap and they fail loudly.

The fix#

The technique that generalises beyond text: count the distinct states, do not compare them.

I spent this week testing generated animations, which have the same problem in a different medium — every one is different and none has a golden form. What I could not assert was "frame 14 looks like this". What I could assert was a structural property:

the frames differ from each other, and the artifact differs from the previous run.

export async function assertVaries(gif: string, previousDigest?: string) {
  const digest = await sha256File(gif);
  // Cross-RUN: byte-identical output means the generator no-opped. This is a
  // structural claim and needs no reference image.
  if (previousDigest && digest === previousDigest) {
    throw new Error(`${gif} is byte-identical to the previous run — nothing regenerated`);
  }

  const dir = await Deno.makeTempDir();
  await run("magick", [gif, "-coalesce", `${dir}/f%03d.png`]);

  const hashes: string[] = [];
  for await (const e of Deno.readDir(dir)) hashes.push(await sha256File(`${dir}/${e.name}`));

  // Cross-FRAME: "it is animated" is a property of the frame set, not of any frame.
  const distinct = new Set(hashes).size;
  if (hashes.length > 1 && distinct === 1) {
    throw new Error(`${gif}: ${hashes.length} frames, 1 distinct — it plays and nothing moves`);
  }
  return { frames: hashes.length, distinct, digest };
}

Two artifacts that both report success from the renderer:

$ assert-varies anim-a.gif
error: anim-a.gif: 36 frames, 1 distinct — it plays and nothing moves

$ assert-varies anim-b.gif
{ frames: 108, distinct: 106, digest: "9f2c1b05…" }

Neither claim needed a reference image, and neither is a comparison against an expected output — which is the only kind of assertion available when every run is legitimately different.

The two assertions cover different failures and neither subsumes the other, which is worth being precise about. Cross-frame variation catches a generator that produced a well-formed artifact with no motion in it — valid file, right duration, nothing happening. Cross-run variation catches a generator that did not run at all and left the previous output in place. A pipeline with only the first will happily re-ship last week's animation; one with only the second will ship a fresh thirty-six-frame still.

What makes both usable is that they are properties rather than comparisons. There is no golden file to maintain, no threshold to tune, and no reviewer deciding whether a difference is acceptable. The cost of that is precision: these assertions cannot tell you the animation is good, only that it is an animation and that it is new. For generated content that is usually the right trade, because the question "is this the expected output" has no answer when the expected output was never specified.

The same shape transfers to text. A generated description cannot be compared against a reference, and it can be asserted to be within a length band, to end on sentence-terminating punctuation, and to differ from the previous generation. Three structural claims, none of them about meaning, and together they catch the clipped, the empty and the stale.

Where this runs out is semantic correctness, and I want to be honest that it runs out early. An animation that varies across frames and across runs can still illustrate the wrong idea entirely, and no structural property will say so. That judgement needs somebody looking — the structural checks exist to make sure the thing being looked at is the thing that was just produced.

That division of labour is the part I would carry to any nondeterministic surface. The machine asserts the properties that have determinate answers; the person judges the thing that does not. The failure mode worth avoiding is the opposite arrangement — a reviewer checking that a file exists and is non-empty, while nothing at all checks whether it is semantically right.

frames    43
distinct  21

At least two distinct states must exist. That is trivially true of a correct animation and immediately false of a broken one, and it required no reference output at all.

The same shape applies to a generated screen. You cannot compare the text. You can require that the rendered result differs from the empty state, differs from the loading state, and has more than one distinguishable configuration across a set of inputs.

How we proved it#

The measurement that made this concrete, from the animation case:

reported     moving: true
measured     41 of 43 frames identical

A converter reported the output was animated, derived from compressed size. Counting distinct frames showed 41 of 43 were the same — technically not static, visually so.

That is the invariant approach catching something a property assertion missed, and the difference is instructive. moving: true is a derived claim. The distinct-frame count is a measurement. When you cannot compare against a reference, measurements survive being wrong about the thing you were deriving.

The trap on the other side, from the same project: a check written to verify that assets recorded their origin searched for the words provenance and procedural. It passed immediately — both already appeared in that file, in unrelated sentences. A check for a topic is not a check for a claim, and on nondeterministic output that mistake is much easier to make, because loose matching is exactly what the problem seems to demand.

The defence is to run the assertion against a known-bad input before trusting it. If it does not go red on something broken, it is not an assertion, it is a description.

The verdict that is not pass or fail#

One structural piece that took me too long to add: a third outcome.

Testing generated output sometimes cannot reach a conclusion — the generation failed, the budget for evaluation ran out, no device was attached. Folding that into "fail" sends someone hunting a defect that does not exist. Folding it into "pass" ships unevaluated work.

The verdict vocabulary here is an enum with unclear in it, and it exists specifically so a judge that is not confident has somewhere honest to put that. An evaluation that exceeds its budget falls back to the cheap static verdict and marks itself unclear rather than pretending the cheap check answered the expensive question.

That value blocks a release exactly as a failure does. What it changes is the next action: attach a device, raise the budget, look by hand — rather than debug code that was never shown to be wrong.

What I have not done#

Being straight about the boundary of this article: the nondeterministic outputs I have actually tested are generated images, animations and prose, not on-device model inference.

The shape of the problem is the same — output varies per run, no golden value, assert invariants — and I believe the technique transfers directly. But I have not run these assertions against Gemini Nano or ML Kit output on a physical device, and there are things I would expect to differ: latency variance affecting what the screen shows mid-stream, partial results rendering during token streaming, device-class differences changing output length enough to matter for layout bounds.

So take the mechanism as tested and the on-device specifics as reasoned. Where I have a number in this article, it came from a real run; where I describe how it would apply to a model on a phone, that is an extrapolation and you should treat it as one.

What this generalizes to#

When output varies per run, stop looking for something to compare and start listing what must be true regardless. Structure present, bounds respected, forbidden states absent, more than one distinguishable configuration reachable.

Then point each assertion at something broken before you trust it, because the failure mode here is not a test that is too strict — it is one so accommodating that nothing can make it red. And give the result a way to say could not determine, so the cases you did not evaluate are visible instead of being quietly counted as passes.