A golden screenshot is a per-platform artifact, whatever the test says

TL;DR#

• A Roborazzi golden set recorded on a developer's Mac and verified in CI on Linux: 312 of 340 screenshots failed, every one of them by a sub-pixel difference in font rasterisation, with nothing behaviourally different.
• A golden image is a function of five things: the renderer, the font files, the font-hinting configuration, the colour space — and your composable. Only the last one is yours.
• The fix is to designate one host as the reference, pin it by digest, and record goldens only there — plus a per-platform set for the two targets whose rendering is genuinely the subject.
• What reading on buys: the four variables measured, why a tolerance threshold is the wrong knob, and the one real regression the noise had been hiding.

The case this post is about#

A screenshot suite over the design system's components:

// desktopTest/kotlin/designsystem/ButtonScreenshotTest.kt
class ButtonScreenshotTest {

    @OptIn(ExperimentalTestApi::class)
    @Test
    fun primaryButton_enabled() = runDesktopComposeUiTest {
        setContent { AppTheme { PrimaryButton("Continue", onClick = {}) } }
        onRoot().captureRoboImage("primary_button_enabled.png")
    }
    // … 339 more
}

Recorded locally with ./gradlew recordRoborazziDesktop and committed — 340 PNG files, in one commit, from one machine.

The symptom#

The same commit, verified in CI:

$ ./gradlew verifyRoborazziDesktop

312 of 340 comparisons FAILED
  primary_button_enabled.png        0.0019 diff (threshold 0.0001)
  primary_button_disabled.png       0.0021 diff
  text_field_focused.png            0.0034 diff
  …

Every diff image looked identical at a glance. Zoomed to 800%, the difference was the antialiasing on glyph edges — a one-level grey shift on the pixels at the boundary of each letter:

# a 4x4 patch from the 'C' in "Continue"
  local (macOS)   0xF2 0xC1 0x3A 0x00 …
  CI (Linux)      0xF1 0xBE 0x37 0x00 …

One to three levels out of 255, on glyph edges only. No layout difference, no colour difference, no missing element. The suite had been green locally and had never run in CI before this commit.

Why the obvious fix failed#

The first fix was to raise the tolerance until CI passed:

RoborazziOptions(compareOptions = CompareOptions(changeThreshold = 0.01F))

340 of 340 passed. And so did a deliberately-broken fixture:

  a button whose corner radius changed from 8.dp to 4.dp     diff 0.0067  → PASSED
  a button whose label changed from "Continue" to "Contnue"  diff 0.0041  → PASSED

A threshold large enough to absorb rasterisation noise is large enough to absorb a visible change. The noise and the signal were the same size, which means the threshold cannot separate them — and raising it had turned 312 false failures into an unknown number of false passes.

The second fix was to record the goldens in a Linux container so CI matched. CI passed, and every developer's local run now failed with the same 312 diffs in the opposite direction. We had moved the problem to the machine where people actually work.

The third attempt was to bundle the fonts so at least that variable was fixed:

val appFont = FontFamily(Font(resource = "fonts/Inter-Regular.ttf"))

Which helped — failures fell from 312 to 94 — and the residual 94 were the useful part, because they proved the fonts were not the only difference.

The actual mechanism#

A rendered image is the output of a pipeline, and four of its five inputs are properties of the host rather than of your code.

The font files. Resolved from the system unless bundled. A Mac and a Linux container do not have the same fonts installed, and the fallback for a missing glyph differs.

The rasteriser and its hinting configuration. Skia on both, with different build flags and different hinting defaults — macOS historically favours stem-darkening and no hinting, Linux distributions commonly enable slight hinting through fontconfig. That is the one- to three-level grey shift.

The graphics backend. The desktop test harness renders through a software Skia path, which is more deterministic than a GPU one and is still compiled per platform with its own flags.

The colour space. A display-P3 default on macOS versus sRGB in a headless container, which shifts non-grey pixels by a level or two.

And only the fifth input — the composable and its theme — is yours.

So a committed PNG asserts a claim about all five, and a test comparing it across hosts is asserting that two different renderers produce byte-identical output, which they do not and are not trying to.

Which is why a tolerance is the wrong knob rather than a poorly-tuned one. The noise is distributed across every glyph edge in the image, so its magnitude scales with how much text the component contains — and a real change to a small element is a smaller diff than the noise on a text-heavy one:

  change                                     diff      noise on the same image
  corner radius 8.dp → 4.dp                  0.0067         0.0019
  a 1.dp padding change on a dense list      0.0008         0.0041     ← noise exceeds signal

There is no threshold that admits the second row's noise and rejects the second row's signal, because the noise is larger.

The fix#

One designated reference host, pinned by digest, with goldens recorded only there — and the two targets whose rendering is genuinely under test keeping their own sets.

// build-logic/src/main/kotlin/screenshot-tests.gradle.kts
//
// Goldens are recorded and verified in ONE pinned container. A screenshot is a function of the
// renderer, the fonts, the hinting config and the colour space, so a golden set is a claim
// about a specific environment — and comparing one host's output to another's asserts
// something neither renderer promises.
//
// Pinned by DIGEST, not by tag. A tag moves, and a moved tag re-records every golden with a
// new rasteriser, which is a 340-file diff nobody can meaningfully review.
val REFERENCE_IMAGE =
    "ghcr.io/example/screenshot-runner@sha256:9f2c1ab4e7d3c08a1f4e6b2d9c7a5083e1b4f6d2a8c3e5071b9d4f6a2c8e3501"

tasks.register<Exec>("recordGoldens") {
    group = "verification"
    description = "Records goldens in the pinned reference container. The ONLY sanctioned way."
    commandLine("docker", "run", "--rm", "-v", "$rootDir:/w", "-w", "/w",
                REFERENCE_IMAGE, "./gradlew", "recordRoborazziDesktop")
}

tasks.register("verifyGoldensLocally") {
    group = "verification"
    doLast {
        // REFUSED rather than run. A local verify would fail for environmental reasons, and a
        // task that fails for environmental reasons teaches people to ignore task failures —
        // measured below. Naming the reason and the alternative is the whole value here.
        error(
            "Golden verification runs in the pinned reference container only.\n" +
            "  Your host's font rasterisation differs from the reference by 1-3 grey levels\n" +
            "  per glyph edge, which fails 312 of 340 comparisons for no behavioural reason.\n" +
            "  Run:  ./gradlew recordGoldens   (to re-record after an intentional change)\n" +
            "  or:   ./gradlew verifyGoldens   (to verify in the container)\n" +
            "  To see what a component looks like on YOUR machine, run the desktop app."
        )
    }
}

// the comparison options — a threshold tight enough to be meaningful
RoborazziOptions(
    compareOptions = RoborazziOptions.CompareOptions(
        // 0.0001 and not 0.01. Inside one pinned environment the output is deterministic, so
        // the threshold only has to absorb genuine nondeterminism (none observed in 50 runs)
        // rather than cross-host rasterisation differences. A loose threshold here absorbed a
        // 4.dp corner-radius change, which is a visible design regression.
        changeThreshold = 0.0001F,
    ),
)

And the two per-platform sets, because for these the rendering is the subject:

// The original 340 split three ways: 298 stay in the desktop reference set, and 42 MOVE OUT
// of it to the platform whose rendering they are actually about. They are not duplicated —
// a golden verified in two environments is the problem this post is about.

// androidUnitTest — Android's own rasterisation, for components with platform-specific
// behaviour: the ripple, the text-selection handles, the system font at a scaled size.
// 23 goldens, recorded on a pinned emulator image.
@Test fun primaryButton_pressed_ripple() { … }

// iosSimulatorArm64Test — CoreText's rendering of the same label, because a text-heavy
// component's line breaking genuinely differs and we ship to iOS.
// 19 goldens, recorded on a pinned simulator runtime.
@Test fun priceLabel_longCurrencyName() { … }

And the output it produced:

$ ./gradlew recordGoldens        # in the pinned container
298 goldens recorded             # 340 minus the 42 that moved to their own platforms

$ ./gradlew verifyGoldens        # CI, same container digest
298 of 298 PASSED
$ ./gradlew verifyGoldensAndroid && ./gradlew verifyGoldensIos
 23 of  23 PASSED
 19 of  19 PASSED

$ ./gradlew verifyGoldensLocally
FAILURE: Golden verification runs in the pinned reference container only.
  Your host's font rasterisation differs from the reference by 1-3 grey levels …

$ # and the broken fixtures, at the tight threshold
  corner radius 8.dp -> 4.dp       diff 0.0067  → FAILED  ✓
  label "Continue" -> "Contnue"    diff 0.0041  → FAILED  ✓
  a 1.dp padding change            diff 0.0008  → FAILED  ✓

How we proved it#

That the diffs were rasterisation and not layout — established pixel by pixel, because 312 failures with identical-looking images is a claim that needs evidence:

  comparison                               differing pixels   max delta   on glyph edges only?
  local vs CI, primary_button_enabled            1,204            3            yes
  local vs CI, text_field_focused                4,881            3            yes
  local vs CI, divider (no text)                     0            0            n/a

The divider — a component with no text — matched exactly, on every host. That is the control, and it localises the whole difference to glyph rendering.

That bundling the fonts fixed part of it. The staged measurement, which is how we learned the fonts were not the only variable:

  configuration                        failures of 340
  system fonts                               312
  + bundled Inter                             94
  + FONTCONFIG_FILE pinned (no hinting)        6
  + the same container digest                  0

Four variables, measured one at a time. Each one is real and none of them is sufficient.

That a tolerance cannot separate noise from signal. The measurement that settled the threshold argument:

  image                      noise (same code, two hosts)   signal (a real 1.dp change)
  a dense list row                   0.0041                        0.0008
  a button                           0.0019                        0.0067

On the first row the noise is five times the signal. There is no single threshold that works for both rows, which means the tolerance knob is answering the wrong question.

That the noise had been hiding a real regression. This is the finding that made the whole exercise worth it — after pinning, one golden failed that had been passing under the loose threshold:

$ ./gradlew verifyGoldens
1 of 298 FAILED
  chip_selected.png   diff 0.0038

# the diff: the selected chip's background had been #2E7D6B and was now #2E7D6C

A one-level colour change from a design-token refactor three weeks earlier, inside the noise floor of the loose threshold and visible at the tight one. It is a trivially small regression and it is exactly the class the suite exists to catch.

That the refusing local task is better than a failing one. Measured by behaviour rather than opinion, over two weeks:

  verifyGoldensLocally RUNS and fails          8 developers ran it, 8 ignored the result,
                                               3 added --continue to their aliases
  verifyGoldensLocally REFUSES with the reason 0 confused runs, 4 used recordGoldens correctly

A task that fails for an environmental reason teaches people to ignore task failures.

What I cannot tell you is how long the pinned container stays valid. It is pinned by digest, so it does not drift — and a JDK or Skia security update will eventually require a re-pin, which re-records all 298 goldens and produces a diff nobody can meaningfully review. The plan is to re-pin deliberately with a visual sweep, and we have not had to do it yet, so the plan is untested.

And an honest cost. A developer cannot see whether their change broke a golden without either running Docker or pushing. We shortened that loop by making recordGoldens fast enough to run locally in the container — about 90 seconds for the suite — and it is still a container, which on a machine without Docker is a blocker.

What this generalizes to#

A golden-image test asserts a claim about an entire rendering environment, most of which is not your code — so it is only a test of your code when the environment is held fixed. Where a comparison is failing for environmental reasons, the instinct to loosen the tolerance is the wrong direction, because environmental noise and the smallest real change are often the same magnitude; pinning the environment is what makes a tight threshold possible, and a tight threshold is what makes the test worth having. And when a check cannot be run correctly in a given place, refusing with the reason beats failing: a check people learn to ignore is worse than no check.

---

Related#

• A Compose UI test in commonTest runs five times and fails on one — the same per-renderer divergence in a text measurement rather than a pixel.
• The same dp is a different number of pixels on every target — why a layout recorded at one density is not a layout.

Keep reading#

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.