The only screenshots worth committing

Two kinds of image come out of a test run. One is the reference a screenshot test compares against; the other is what the run actually produced, plus any diffs.

They look identical in a file browser, and exactly one of them belongs in version control.

The case this post is about#

Two kinds of PNG in one repository, and only one of them belongs in version control.

A golden render reference is an input: the image a test compares against, which has to be reviewed when it changes and is meaningless if it drifts silently. A run capture is evidence: regenerable, dated, and worth exactly nothing a week later.

Before the split, both were committed. The repository carried hundreds of device captures from runs nobody remembered, every one of them a binary that no diff can show, and the handful of real goldens were indistinguishable from them.

This post follows that distinction, because "commit your screenshots" is advice that is half right, and the half that is wrong turns a repository into a photo album.

The symptom#

The repository gets heavy. Image files accumulate in test output directories, every run adds more, and a clone takes noticeably longer than it should for the amount of code involved. Somebody adds an ignore rule, which helps until a directory layout changes and the rule stops matching.

The second thing is worse and quieter: the golden references get lost among them. If run captures and references live in the same place, a cleanup that deletes generated images takes the references with it, and the next test run regenerates references from current behaviour — which means the test now asserts that the screen looks the way it currently looks, whatever that is. The suite stays green through the exact change it existed to catch.

The third is review noise. A pull request that touches one screen arrives with forty changed images, most of them run artifacts from an unrelated test. The two or three that are genuine reference updates — the ones a human should actually look at — are buried, and the reviewer approves the batch because reviewing forty images individually is not a thing anybody does.

The fourth is the merge conflict. Binary files do not merge. Two branches that both ran the suite produce conflicting captures in files nobody edited, and resolving them is arbitrary because neither version means anything.

What makes that last one corrosive rather than merely annoying is what it teaches. Resolving a meaningless binary conflict is done by picking a side without looking, several times a week, until picking a side without looking is simply how image conflicts are handled — including the one, months later, that is a real reference and did mean something.

Why the obvious fix failed#

Ignore all images. Simple and it throws away the references, which are a real input the test needs. A screenshot test with no committed reference regenerates it on first run and then passes forever, which is the failure above arrived at deliberately.

Commit everything and let the repository grow. Defensible for a while and it degrades continuously. Clone time, checkout time and review noise all get slightly worse with every run, and no single run is responsible, so there is never a moment where somebody decides to act.

Store images externally. Solves the size problem and introduces a synchronisation one: the references now live somewhere the repository does not control, so a checkout of an old commit does not get the references that commit was tested against. Version control exists precisely to make that correspondence hold.

Clean up periodically. A manual sweep, which works until the sweep deletes a reference by mistake — and that mistake is silent, because the next run recreates it from current output and everything passes.

The common thread across all four is that each one treats this as a storage problem. It is not: it is a classification problem that shows up as a storage problem, and every fix aimed at the symptom either keeps both kinds of file or discards both. The size of the repository is the part people notice, and it is the least important of the three consequences.

The actual mechanism#

Classify by whether the file is an input or an output, and let that decide everything else.

golden reference   an INPUT. The test compares against it.        → committed
run capture        an OUTPUT. Evidence of what happened.          → ignored
diff image         an OUTPUT. Derived from the other two.         → ignored

A reference is not a screenshot in any meaningful sense — it is a test fixture that happens to be an image. It is reviewed when it changes, it is the thing a failure is measured against, and a repository without it cannot run the test. Everything else is regenerable from the code and the reference, which is the definition of something that does not belong in version control.

The classification has to be expressed as paths, not patterns. A rule that ignores images by extension catches references too; a rule that ignores a directory keeps the distinction structural, so the only way to commit a capture is to put it somewhere it does not belong. That makes the mistake visible in review rather than invisible in a glob.

The sub-case that matters most is reference updates. When a screen legitimately changes, its reference must change too, and that commit is the one a human should look at properly — it is a statement that the new appearance is correct. Keeping captures out of version control is what makes that possible: a pull request containing three changed references and nothing else is reviewable, and one containing three references and forty captures is not.

There is a related discipline about where references live. Alongside the test that uses them, in the module that owns the screen — not in a central image directory, which turns a question about one screen into a question about a shared folder and makes ownership ambiguous.

That placement decision also makes deletion behave correctly. A screen that is removed takes its test and its references with it, because they are in the same place and nobody has to remember the third one. A central directory accumulates references for screens that no longer exist, and because nothing reads them, nothing reports them — so the only evidence is a folder that slowly grows and is never audited.

The fix#

references   committed, beside the test, in a declared directory
captures     written to an output path that is ignored by structure
diffs        same output path, same treatment
review       a reference change is a real review, in isolation

The layout decides it. References sit beside the test and are committed; everything a run produces goes to one ignored directory — so the ignore rule is structural rather than a list of filenames somebody maintains:

feature/profile/src/commonTest/
  roborazzi/                      <- COMMITTED. the references.
    ProfileScreen_empty.png
    ProfileScreen_loaded.png
build/roborazzi/                  <- ignored, wholesale. captures AND diffs.

@Test
fun profileScreen_loaded() {
    captureRoboImage(
        filePath = "src/commonTest/roborazzi/ProfileScreen_loaded.png",
        roborazziOptions = RoborazziOptions(
            compareOptions = RoborazziOptions.CompareOptions(
                // A threshold above zero is a threshold somebody will raise under
                // deadline. Zero means a diff is a decision, never a negotiation.
                changeThreshold = 0.0f,
                outputDirectoryPath = "build/roborazzi",   // captures + diffs, never beside the ref
            ),
        ),
    ) { ProfileScreen(state = loadedState, onSave = {}) }
}

# one structural rule, not a per-file list that drifts
build/roborazzi/

A padding change nobody mentioned in the description:

$ ./gradlew verifyRoborazziDebug
✗ ProfileScreen_loaded.png  0.31% of pixels differ
    reference  src/commonTest/roborazzi/ProfileScreen_loaded.png
    actual     build/roborazzi/ProfileScreen_loaded_actual.png
    diff       build/roborazzi/ProfileScreen_loaded_compare.png
FAILED

$ ./gradlew recordRoborazziDebug        # accepting it
  1 reference updated — commit it; the diff IS the review

The committed reference is the only screenshot in the repository. Captures and comparison images are regenerable evidence and live where git status never mentions them.

The last line is the point of all of it. Everything above exists so that a changed reference arrives in a reviewable state and means something — because a reference update is somebody asserting that a visual change is intended, and that assertion is worthless if it is buried in noise or applied automatically.

How we proved it#

The silent-regeneration failure is reproducible, and it is what justifies the whole distinction: delete a reference, run the suite, and the test passes while generating a new reference from current output. Nothing warns, nothing fails, and the test has quietly become a tautology. That is a one-minute demonstration and it is the strongest argument for treating references as committed inputs.

The structural ignore rule is checkable. Either captures are written under an ignored path or they are not, and git status after a test run answers it. A clean status after a full run is the property you want, and it is binary.

That check is worth wiring into the build rather than remembering, because it is the thing that decays. The paths are correct on the day somebody sets them up; what breaks them is a framework upgrade that changes a default output directory, or a new test written by somebody copying an older one from before the convention. Neither announces itself, and both are caught by the same two-second assertion that the working tree is clean after a run.

What I have not measured is repository size before and after. The argument here is about the reference-loss failure and about review legibility rather than about bytes, and I did not take a baseline — so I cannot tell you what the captures were costing in clone time.

What I have not solved: reference review at scale. A changed reference should be looked at, and looking at it means a human comparing two images and deciding whether the difference is intended. With three references that is fine; with sixty — after a design system change that legitimately alters every screen — nobody genuinely inspects all of them, and the approval becomes a formality. I do not have an answer better than keeping such changes rare.

And a limitation of golden tests generally that this does not address: they are sensitive to things that do not matter. A font rendering difference, an antialiasing change, a platform update — all produce diffs that are real at the pixel level and meaningless at the behaviour level. The classification in this post makes the references manageable; it does nothing about the flakiness that makes some teams abandon the technique.

The regeneration flag is where I would warn somebody specifically. Every screenshot test framework offers a way to accept current output as the new reference, because without one a legitimate design change would be unworkable. That flag is also the exact mechanism by which the test becomes a tautology, and the distance between "update the references, the redesign landed" and "update the references, the tests are red" is a single sentence in a commit message.

I do not think the flag should be removed; I think its use should be visible. A reference update that arrives in its own commit, touching only references, is a statement somebody made. One that arrives inside a larger change, alongside the code that altered the appearance, is the same bytes with none of the meaning — and the reviewer is now being asked to verify both the change and its own evidence at the same time.

What this generalizes to#

Inputs and outputs look the same on disk, and version control only makes sense when you have decided which is which.

The confusion recurs everywhere a tool reads and writes the same kind of file: a lockfile versus a resolution cache, a schema versus a generated migration, a baseline versus a report, a fixture versus a recording. In each case one artifact is the thing the system is held against and the other is what the system produced, and in each case committing the second produces noise while losing the first removes the check entirely.

The test is whether the file can be regenerated from the repository as it stands. If it can, it is an output and committing it adds nothing. If it cannot — if the repository without it is missing information that no amount of running will reconstruct — it is an input, and it deserves a reviewed commit every time it changes.