How to know your on-device model got worse

A model that gets slower throws nothing. A model that gets worse throws nothing either.

Every other regression in a build announces itself — a test fails, a type breaks, a crash reports. Quality is the one that degrades silently and ships, because nothing in the pipeline is looking at the content of what came out.

The case this post is about#

One twelve-case suite, run against a $0.40 judging budget:

case-03  stub-behavior   static
case-06  partial         judged ($0.06)
case-10  unclear         budget exhausted ($0.42)
case-11  unclear         budget exhausted ($0.42)
case-12  unclear         budget exhausted ($0.42)

Three cases were never judged. They are not passes and they are not failures — the question was not asked, because the money ran out three cases from the end.

A suite with a two-value vocabulary has to call them something. Green is a lie; red sends somebody to debug working code. This post follows those three rows, because partial on case-06 blocks for the opposite reason — it was judged, and the answer was neither.

The symptom#

The model changes. A new version, a different quantisation, a platform update underneath it, a prompt edit three files away. The app builds, the tests pass, the screen renders.

What you cannot see from any of that is whether the output got worse — and "worse" has several shapes, each of which needs a different check:

• answers that became wrong
• answers that became vaguer while staying technically correct
• answers that became longer and broke a layout that assumed brevity
• answers that stopped following a format the parser depends on

The last one is the only one a conventional test catches, and it is the least interesting. The others produce a working app that is quietly less useful, and the usual way anyone finds out is a complaint.

Why the obvious fix failed#

The obvious fix is a large evaluation set. Hundreds of cases, scored carefully, run before release.

It fails on cadence, not on quality. A suite that takes an hour runs weekly at best, which means a regression is discovered a week's worth of changes after it was introduced and the bisect is expensive. The suite is good and the feedback loop is useless.

The second attempt is to score with another model, which works and introduces a cost that scales with the suite — and an expensive check that sometimes gets skipped is a check whose results you cannot reason about, because you do not know which runs had it.

The actual mechanism#

A small suite that runs on every build and fails it.

Small is the design constraint, not a compromise. Ten to twenty cases, chosen because each one dies in a distinguishable way, running fast enough that nobody is tempted to make it optional.

What each case asserts is the technique from testing any generated output: invariants, not strings.

• the output parses into the shape the app consumes
• required fields are present and non-empty
• length falls inside the bounds the layout was designed for
• forbidden content is absent — refusals, placeholders, error sentinels
• for cases with a knowable answer, the answer is correct

Only the last of those needs judgement. The rest are mechanical, which is what makes the suite cheap enough to run every time.

The fix#

Two structural pieces matter more than the case list.

Bound the expensive part with a budget. Where a case needs a model to judge another model's output, that judgement has a cost ceiling. Past it, the run falls back to the cheap static checks and marks itself accordingly rather than pretending the cheap checks answered the expensive question.

The budget is a number in config rather than a code path, which matters more than it sounds: a ceiling you can raise for one run is a ceiling people leave in place, and a ceiling buried in a conditional is one somebody deletes the first time it is inconvenient. Expressed as a value, the question "was this run's judgement bounded, and by how much" has an answer you can read off the run rather than infer from the code.

Give the verdict a third value. The outcome vocabulary needs something that is neither pass nor fail:

real-behavior | stub-behavior | partial | unclear

unclear exists so a judge with low confidence has somewhere honest to put it.

Both pieces are one runner, and the thing worth noticing is that the budget and the vocabulary interact — an exhausted budget produces unclear, never a pass:

type Verdict = "real-behavior" | "stub-behavior" | "partial" | "unclear";
type Case = { id: string; prompt: string; staticCheck: (out: string) => boolean; needsJudge: boolean };

async function runSuite(
  cases: Case[],
  infer: (p: string) => Promise<string>,
  judge: (p: string, out: string) => Promise<{ verdict: Verdict; costUsd: number }>,
  budgetUsd: number,
) {
  let spent = 0;
  const results: Array<{ id: string; verdict: Verdict; how: string }> = [];

  for (const c of cases) {
    const out = await infer(c.prompt);
    const staticPass = c.staticCheck(out);

    if (!c.needsJudge) {
      results.push({ id: c.id, verdict: staticPass ? "real-behavior" : "stub-behavior", how: "static" });
      continue;
    }
    if (spent >= budgetUsd) {
      // The cheap check did NOT answer the expensive question. Saying so is the
      // whole point of the third value — a budget-exhausted run must not read as green.
      results.push({ id: c.id, verdict: "unclear", how: `budget exhausted ($${spent.toFixed(2)})` });
      continue;
    }
    const j = await judge(c.prompt, out);
    spent += j.costUsd;
    results.push({ id: c.id, verdict: j.verdict, how: `judged ($${j.costUsd.toFixed(2)})` });
  }

  const tally = results.reduce((m, r) => (m[r.verdict] = (m[r.verdict] ?? 0) + 1, m), {} as Record<Verdict, number>);
  // Only real-behavior is green. partial and unclear both BLOCK, for different reasons.
  return { results, tally, spentUsd: Number(spent.toFixed(2)),
           green: (tally["stub-behavior"] ?? 0) === 0 && (tally.partial ?? 0) === 0 && (tally.unclear ?? 0) === 0 };
}

A 12-case suite with a $0.40 ceiling, where nine of the twelve need a judge at roughly $0.07 each:

case-01  real-behavior   static
case-02  real-behavior   static
case-03  stub-behavior   static
case-04  real-behavior   judged ($0.07)
case-05  real-behavior   judged ($0.07)
case-06  partial         judged ($0.06)
case-07  real-behavior   judged ($0.07)
case-08  real-behavior   judged ($0.07)
case-09  real-behavior   judged ($0.08)
case-10  unclear         budget exhausted ($0.42)
case-11  unclear         budget exhausted ($0.42)
case-12  unclear         budget exhausted ($0.42)

tally { real-behavior: 7, stub-behavior: 1, partial: 1, unclear: 3 }
spent $0.42   green false

Three cases were never judged and they are not passes. A suite with a two-value vocabulary has two options for those rows — call them green, which is a lie, or call them failures, which sends somebody to debug working code — and both are worse than a row that says the question was not asked. The partial on case-06 blocks for the opposite reason: it was judged, and the answer was neither. A run that exceeded its budget, could not reach a device, or got an ambiguous result lands there — and that value blocks a release exactly as a failure does while telling whoever reads it to do something different: raise the budget, attach a device, look by hand.

Collapse that into "fail" and someone hunts a regression that does not exist. Collapse it into "pass" and you ship work nothing evaluated.

How we proved it#

Being precise about what I have actually run, because this is the article where it matters most.

Measured: the verdict structure and the budget bound are real and in use here, scoring generated content rather than model inference. The unclear value earns its place regularly — most often when an evaluation could not be completed rather than when a judgement was close.

Measured: the invariant-assertion technique, against generated animations and images. The concrete case is a converter that reported moving: true while 41 of 43 frames were identical — a derived verdict that was wrong, caught only because the underlying count was also recorded.

That second one is the transferable lesson for model quality: log the measurement, not just the score. A quality number with no underlying counts cannot be audited when it drifts, and quality numbers drift for boring reasons — a changed prompt, a different sampling temperature — that you can only separate from real regressions if the components are recorded.

Not measured: I have not run an eval suite against Gemini Nano, ML Kit or Core ML on a device. The suite design, the budget bound and the verdict vocabulary are from scoring non-inference output, and I am extrapolating the on-device application.

What I expect to be harder on a device#

Three things I would watch for, stated as expectations rather than findings:

Device variance is a confound. The same model on two handsets may produce different output, so a suite that fails on one device and passes on another is reporting something real and not a regression. A single device in CI measures that device.

Thermal state affects results. A phone that has been running a suite for two minutes is not in the state it was in at case one, which means ordering can matter and a flaky case may be a thermal artifact rather than a quality one.

The baseline moves underneath you. A platform model updates outside your release cycle, so a regression can appear in a build that changed nothing relevant. That makes it essential to record which model version produced each result — otherwise the first thing you will do is bisect your own code for a change that is not there.

All three argue for the same thing: record the conditions alongside the score. Which device, which model version, what position in the run. Without those, a failing eval tells you something is different and leaves you to guess what.

What this generalizes to#

For anything whose output quality can degrade without throwing, build the smallest suite that fails the build, and keep it small deliberately — a check that runs every time beats a thorough one that runs weekly.

Assert invariants rather than strings, bound any expensive judgement with a budget, and give the verdict somewhere to say could not determine so the unevaluated cases stay visible. Then record the measurements and the conditions beside every score, because a quality number that drifts is useless unless you can tell which of its inputs moved.