An agent that installs the app and looks at the screen

An agent that can build an app, install it, and look at the result closes a loop that otherwise needs a person.

The loop is five steps, it runs in one order, and every step is there because the capture is worthless without it.

The case this post is about#

One capture loop, five steps, run after every build:

pkg="com.example.storefront"
apk="app/build/outputs/apk/debug/app-debug.apk"
# 1. The APK ON THE DEVICE must be the one just built. `install -r` is not enough —
#    a failed install leaves the old APK and the next step screenshots it happily.

The steps are: install, force-stop, hash-match the device APK against the built one, capture, and assert the captured screen is the route under test.

Drop any one of them and the loop still produces a PNG, still exits 0, and still reports success. That is what makes the order worth writing down rather than remembering — the output of a correct run and the output of a run missing a step are the same file, and only one of them is evidence.

The symptom#

The naive loop is two steps: install, screenshot. It produces an image, the image looks like evidence, and it answers a different question than the one asked.

What makes it slippery is that the failure is invisible in the artifact. A capture taken from a surviving process, a capture of the wrong route, a capture of a build that never installed — all three look exactly like a correct one. There is no visual difference between a screenshot that proves your change works and a screenshot that proves nothing at all.

So the loop cannot be validated by looking at its output. It has to be validated by its procedure, which means the procedure has to be written down and followed rather than reconstructed each time.

Why the obvious fix failed#

The obvious fix is to add a wait. Install, sleep, screenshot — give the app time to come up.

It converts a reproducible wrong answer into an intermittent one, which is worse. The race it papers over is not a timing bug; it is that nothing in the sequence forced the new build to be the thing on screen.

The second attempt is to trust the install output. Installed on 1 device is a true statement about where an APK landed and says nothing about which process is drawing pixels.

The actual mechanism#

Five steps, fixed order:

adb shell am force-stop <pkg>
./gradlew :app:installDebug
adb shell monkey -p <pkg> -c android.intent.category.LAUNCHER 1
adb exec-out screencap -p > shot.png
adb shell md5sum $(adb shell pm path <pkg> | sed 's/^package://')

Each earns its place by a failure it prevents:

Force-stop first. Not politeness — it is what guarantees no surviving process can render an older composition. This is the step that looks skippable and is the reason the loop exists.

Install second, so the stopped app is replaced rather than the running one patched.

Launch explicitly, rather than assuming the install started anything. An explicit launcher intent makes the app's start a step you performed rather than one you hoped for.

Capture fourth, when the only process that can be drawing is the one you just started.

Hash last, comparing the on-device APK against the one you built. The screenshot carries no build identity of its own; this is where it gets one.

The fix#

The step I keep wanting to add and have not is an assertion on content, not just on process.

A capture taken correctly, of the wrong screen, passes every check above. The hash matches, the process is fresh, the image is of a route your change never touched. The loop proves the build is right and says nothing about the screen.

The cheap version of the missing step is to require a known identifier from the route under test to be present in the captured view hierarchy — a test tag that only that screen emits. Then the loop asserts "this is the right screen, freshly drawn, from the build I made", which is three claims rather than one.

I have this for the automated flows and not for the ad-hoc captures, which is exactly backwards: the ad-hoc ones are where a human is eyeballing a screenshot and most likely to accept it.

The five steps as one function, with the content assertion in the place it belongs — before the capture is returned, not after somebody has looked at it:

capture_evidence() {
  local pkg="$1" route_tag="$2" apk="$3" out="$4"

  # 1. INSTALL. -r does not kill a running process, which is the whole trap.
  install_apk "$apk" || return 1

  # 2. STOP THE APP. Without this the screenshot shows the OLD composition, under
  #    a build that is genuinely new -- the two facts are independently true.
  stop_app "$pkg"

  # 3. PROVE the device is running the build you just made. A hash comparison,
  #    not an assumption that the install succeeded.
  local built device
  built=$(digest_of "$apk")
  device=$(digest_of_installed "$pkg")
  if [ "$built" != "$device" ]; then
    echo "device has $device, built $built — NOT the same artifact" >&2
    return 2
  fi

  # 4. CAPTURE, after a relaunch, so the composition is drawn fresh.
  launch_route "$pkg" "$route_tag"
  screenshot_to "$out" || return 3

  # 5. ASSERT THE SCREEN. A perfect capture of the wrong route passes 1-4.
  if ! view_hierarchy "$pkg" | contains_tag "$route_tag"; then
    echo "capture is valid but shows the wrong screen: no \"$route_tag\" in the hierarchy" >&2
    return 4
  fi

  echo "evidence: $out — route $route_tag, build $built, freshly drawn"
}

The same change captured four ways, three of which look like success:

$ capture_evidence com.example.app agentic_state_tab app-debug.apk cap.png
evidence: cap.png — route agentic_state_tab, build 0c57da24, freshly drawn

# step 2 omitted
$ capture_evidence ...
evidence: cap.png — route agentic_state_tab, build 0c57da24, freshly drawn
  (the image shows the PREVIOUS composition; nothing here says so)

# step 3 omitted, stale build on device
$ capture_evidence ...
evidence: cap.png — route agentic_state_tab, build 0c57da24, freshly drawn
  (the image is of a build from yesterday)

# all five steps, wrong route launched
$ capture_evidence com.example.app agentic_state_tab app-debug.apk cap.png
capture is valid but shows the wrong screen: no "agentic_state_tab" in the hierarchy
exit 4

The two middle runs print the same success line as the first, which is why the steps have to be in the function rather than in a checklist — an omitted step produces evidence that is indistinguishable from real evidence. And the last run is the step I said I had not generalised: it is the only one of the four failures that a human looking at a plausible screenshot would not have accepted.

How we proved it#

The direct test is whether the loop and the naive version disagree on a build where the change is genuinely present.

Without the force-stop:

fix visible: NO

Same device, same APK, with it:

fix visible: YES

The code was correct throughout. The first capture photographed a process that started before the change existed.

And the hash step, on a run where the install had silently not replaced the package:

device: 4a9f2c1e8b7d3f06a5c2e91b4d7a8f30
built:  b7e14d09fa62c8531e0d4a7b29c6f815

Different hashes. Every screenshot from that session was of a build nobody had on disk, and nothing in any of those images would have said so.

What the loop cannot see#

Three limits, and the third is the one that matters most for an agent.

It cannot judge the screen. The loop proves a fresh capture of a known build. Whether the layout is correct — spacing right, states rendering, the designed thing actually present — is a judgement the loop delivers an image for and does not make.

It cannot run without a device. No phone attached means the honest outcome is not a pass and not a failure, but a third state: undetermined. Folding that into either of the other two is how a release ships untested or a team hunts a bug that does not exist.

It confirms rendering, not behaviour. A screen can draw perfectly and do nothing when tapped. The capture loop and an interaction test are different instruments, and a corpus of beautiful screenshots is a well-known way to feel verified while every handler is empty.

When there is no phone#

The undetermined case deserves more than a mention, because it is the one an unattended run meets most often and the one most likely to be papered over.

The fallback is to spin an emulator, and it needs a deadline — here, five minutes to boot. Without one, a run that cannot find a device sits waiting for a virtual one that is not coming, and an agent waiting is indistinguishable from an agent working.

Past the deadline the correct outcome is not failure. It is a distinct verdict meaning the test exists and could not be executed, which blocks a release exactly as a failure would while telling whoever reads it to attach a device rather than to debug the code.

Collapsing that into "failed" sends someone hunting a bug that does not exist. Collapsing it into "passed" ships untested work. The third state is not pedantry; it is the difference between two opposite next actions.

Why this belongs to the agent rather than a person#

The reason to automate this is not speed, it is that the order is exactly the kind of thing people optimise away.

Every step in that sequence looks skippable in the moment. The force-stop feels like tidiness. The explicit launch feels redundant after an install. The hash feels paranoid on a run where nothing seemed wrong. A person under time pressure drops one, gets away with it four times, and is misled on the fifth.

A script does not get bored of the force-stop. That is the whole argument — not that the agent judges better, but that the sequence it runs is the sequence that was designed, every time, including the times nobody would have checked.

What this generalizes to#

When an agent inspects a real system to confirm a change, write the capture procedure down as an ordered sequence and treat each step as a precondition rather than a courtesy. Then ask what the artifact would look like if the step you most want to skip had been skipped — if the answer is "identical", that step is the one holding the whole thing up.

And give the result somewhere to be undetermined. A loop that can only pass or fail will eventually report one of them about a run that did neither.