A flaky test is a timing assumption you didn't know you made

Day 86 of 100. Flakiness has one dominant cause in Compose tests, and one family of fixes.

The case this post is about#

One test that passes locally and times out in CI.

It asserts that a list renders after loading. Locally the data resolves from a warm fake in a few milliseconds and the assertion runs after everything has settled. In CI the same test waits five seconds and fails, having never reached the assertion at all.

Nothing about the list is slower in CI. What differs is that the screen contains an indeterminate progress indicator, and an infinite animation is never idle — so the framework's automatic wait, which is what makes most Compose tests need no waiting, has nothing to wait for.

This post follows that one test, because the fix is not a longer timeout: it is taking the clock, so the test advances time itself rather than waiting on a condition defined never to hold.

The symptom: a test that passes locally and fails in CI#

A test that passes locally and fails in CI:

@Test fun ordersLoad() = runComposeUiTest {
    setContent { OrderScreen(viewModel) }
    Thread.sleep(500)                                   // "give it time to load"
    onNodeWithText("Order #4711").assertIsDisplayed()
}

On a fast machine, 500ms is plenty. On a loaded CI runner it sometimes isn't, so the test fails perhaps one run in twenty. Someone raises the sleep to 1500ms, the flakiness becomes rarer, and the suite gets three seconds slower per test.

Nobody has fixed anything; the failure has been made less frequent.

Why longer sleeps and @FlakyTest only hide it#

The obvious fix is a longer sleep, then a retry rule, then @FlakyTest.

Each step makes the signal worse. A retried test hides a real intermittent bug just as effectively as it hides a timing artefact, and a suite with retries stops being evidence about anything.

More to the point, the sleep was usually unnecessary. Compose tests synchronise automatically, and the sleep was covering for something the framework couldn't see.

The mechanism: the test framework waits until the composition is idle#

Before every finder and assertion, the test framework waits until the composition is idle — no pending recompositions, no pending layout, no pending draw, and no running animations.

That's why most tests need no waiting:

@Test fun ordersLoad() = runComposeUiTest {
    setContent { OrderScreen(fakeViewModel) }
    onNodeWithText("Order #4711").assertIsDisplayed()     // already synchronised
}

The assertion doesn't run until the composition settles. A Thread.sleep before it is at best redundant.

Idleness is tracked through a set of registered IdlingResources, and the framework registers the Compose ones for you. Everything below is about the cases where that tracking has a gap or never completes.

Idleness is a conjunction over registered resources, and a conjunction is exactly where a never-settling one hides:

interface IdlingResource { val name: String; val isIdle: Boolean }

class TestClock {
    var now = 0L; private set
    private val resources = mutableListOf<IdlingResource>()
    fun register(r: IdlingResource) { resources += r }

    // The gate every finder passes through. One busy resource holds all of them.
    fun waitForIdle(timeoutMs: Long = 5_000, stepMs: Long = 16): Result<Long> {
        val start = now
        while (now - start < timeoutMs) {
            val busy = resources.filter { !it.isIdle }
            if (busy.isEmpty()) return Result.success(now - start)
            now += stepMs                       // advance the clock, do not sleep
        }
        return Result.failure(IllegalStateException(
            "idle timeout after ${timeoutMs}ms; busy: " +
                resources.filter { !it.isIdle }.joinToString { it.name }))
    }
}

Two screens, same test harness — a loading spinner that stops, and one that does not:

val clock = TestClock()
clock.register(object : IdlingResource {
    override val name = "recomposer"
    override val isIdle get() = clock.now >= 48          // settles after 3 frames
})
clock.register(object : IdlingResource {
    override val name = "InfiniteProgressIndicator"
    override val isIdle get() = false                     // an infinite animation NEVER settles
})
println(clock.waitForIdle())

Failure(java.lang.IllegalStateException: idle timeout after 5000ms;
        busy: InfiniteProgressIndicator)

With the indicator removed from the registration, the same call returns Success(48) — three frames. The five-second difference is the whole diagnosis, and the message naming the resource is what turns "the test hangs" into "this animation never stops". That is also why mainClock.autoAdvance = false is the fix rather than a longer timeout: taking the clock lets you advance 48ms and assert, instead of waiting for a condition that is defined never to hold.

Cause 1: something is never idle#

The most common, and Day 66's subject: an infinite animation means the composition never reports idle, so every implicit wait times out.

mainClock.autoAdvance = false

With the clock stopped, the animation advances only when you say so, and idle-waiting stops being relevant. Then step time explicitly:

mainClock.advanceTimeByFrame()    // let one frame happen
mainClock.advanceTimeBy(300)      // past an animation

A shimmer placeholder, a pulsing dot or a progress spinner on the screen under test is enough to trigger this, and the timeout message never mentions animation — which is why it costs people an afternoon the first time.

Cause 2: work the framework can't see#

Compose knows about its own recomposition. It doesn't know about your coroutine hitting a network, a Handler.postDelayed, or a background thread.

Two fixes, in order of preference:

Make the work synchronous in tests. An injected test dispatcher is the honest approach:

@Test fun ordersLoad() = runComposeUiTest {
    val dispatcher = StandardTestDispatcher()
    val viewModel = OrderViewModel(repo = FakeRepo(), dispatcher = dispatcher)
    setContent { OrderScreen(viewModel) }

    dispatcher.scheduler.advanceUntilIdle()      // run the pending coroutines
    onNodeWithText("Order #4711").assertIsDisplayed()
}

No waiting at all — the work completes deterministically because the test controls when coroutines run. This is the fix that removes flakiness rather than reducing it.

Register an idling resource, when the work genuinely can't be made synchronous:

val resource = object : IdlingResource {
    override val isIdleNow: Boolean get() = !repository.hasInFlightRequests
}
composeTestRule.registerIdlingResource(resource)

Now the framework's automatic waiting includes your work, and the tests stay sleep-free.

The same mechanism is what makes Espresso and Compose tests interoperate: both consult the registry, so an idling resource registered for a Retrofit call serves whichever framework is asserting. In a mixed codebase that's worth knowing before writing a second one.

Cause 3: a genuinely asynchronous wait#

Sometimes you're testing something that legitimately takes an unknown time — an integration test against a real service, say. waitUntil is the tool, and it's still better than a sleep:

composeTestRule.waitUntil(timeoutMillis = 5_000) {
    onAllNodesWithText("Order #4711").fetchSemanticsNodes().isNotEmpty()
}

It polls a condition and returns as soon as it's true, so it's fast in the common case and only slow when something is wrong. waitUntilExactlyOneExists, waitUntilAtLeastOneExists and waitUntilDoesNotExist are the readable shorthands for the common conditions.

A waitUntil in a tier-2 test is a smell — it means the test isn't controlling its dependencies. In a tier-3 integration test it's legitimate.

The rule#

In order:

1. Nothing. Most tests need no synchronisation at all.
2. A test dispatcher, if the screen does asynchronous work.
3. mainClock.autoAdvance = false, if animation is involved.
4. An idling resource, if there's background work you can't control.
5. waitUntil, for genuinely unknown timing.
6. Thread.sleep — never. There is no case where it's the right answer.

Two smaller traps#

waitForIdle() doesn't help with a never-idle screen. It's the explicit form of the automatic wait, so it has the same failure mode. Reaching for it when a test times out is the natural move and the wrong one; the clock is the fix.

State changed from the test thread needs a frame. Setting a mutableStateOf directly in a test schedules a recomposition rather than performing one:

visible = false
mainClock.advanceTimeByFrame()      // needed when autoAdvance is off
onNodeWithTag("banner").assertDoesNotExist()

With autoAdvance on this is handled; with it off, the missing frame advance produces a test that asserts against the previous state and fails confusingly.

How to prove it: run the suspect test two hundred times#

The direct test for flakiness is repetition. Run the suspect test two hundred times:

./gradlew :app:testDebugUnitTest --tests "*.OrderScreenTest" --rerun-tasks

A timing-dependent test fails somewhere in that run. A properly synchronised one doesn't, because there's no timing left for it to depend on.

The cheaper signal: grep the suite for Thread.sleep and waitUntil. Every hit is a place where the test is guessing, and each one is a candidate flake.

A lint rule banning Thread.sleep in test sources is worth adding once. It's the kind of thing that creeps back in under deadline pressure, and a build failure is a cheaper conversation than a flaky suite six months later.

What this generalizes to: time is an input, and determinism means controlling it#

The principle is determinism comes from controlling inputs, and time is an input — Day 66's line, and it holds across the whole testing pillar.

A test that waits is a test whose result depends on the machine it ran on. Every technique above replaces a wait with control: a test dispatcher controls when coroutines run, the test clock controls when frames happen, an idling resource makes hidden work visible to the framework. The flakiness disappears not because the waiting got better but because there is nothing left to wait for.

It's worth noticing that this is the same move as Day 62's animation argument, one level up. There, replacing a scripted journey with a declared target removed the need to handle interruption. Here, replacing an elapsed-time assumption with an explicit clock removes the need to handle slowness. Both work by deleting the thing that could vary rather than by accommodating it.

Tomorrow, Day 87 closes the testing pillar with the patterns — what a good Compose test suite actually contains.

---

Day 86 of a 100-day series on Jetpack Compose, working through the official documentation in order. Source: Testing synchronisation.