runTest skips your delays, which is why the flaky test passed

TL;DR#

runTest runs on a TestDispatcher with a virtual clock. delay(30.seconds) returns immediately; the clock jumps instead of the thread waiting. That is why coroutine tests are fast and deterministic, and it is the right default.

It also means two things you will eventually be bitten by. A test can assert the effect of a timeout without ever proving the timeout fires in real time. And any coroutine that escapes onto a real dispatcher is no longer on the virtual clock, so advanceUntilIdle() returns while that work is still in flight — producing a test that passes locally, fails on CI, and gets labelled flaky.

advanceUntilIdle advances virtual time. It is not "wait for everything".

The case this post is about#

A sync manager with a timeout and a retry, under test:

// commonMain
class SyncManager(private val api: Api, private val dispatcher: CoroutineDispatcher) {

    suspend fun sync(): SyncResult = withContext(dispatcher) {
        repeat(3) { attempt ->
            try {
                return@withContext withTimeout(30.seconds) { api.pull() }
            } catch (e: TimeoutCancellationException) {
                delay(2.seconds * (attempt + 1))        // backoff
            }
        }
        SyncResult.Failed
    }
}

// commonTest
@Test
fun retriesThreeTimesThenFails() = runTest {
    val api = HangingApi()
    val result = SyncManager(api, Dispatchers.Default).sync()   // ← the problem
    assertEquals(SyncResult.Failed, result)
    assertEquals(3, api.attempts)
}

This test passes. It finishes in 40 milliseconds despite describing 96 seconds of behaviour, which should have been the clue.

The symptom#

The test passed locally, every time, and failed on CI about one run in six:

retriesThreeTimesThenFails   PASSED   (local, 41ms)
retriesThreeTimesThenFails   PASSED   (local, 38ms)
retriesThreeTimesThenFails   FAILED   (CI)
    expected 3 attempts but was 1
retriesThreeTimesThenFails   FAILED   (CI)
    expected 3 attempts but was 2

Two details made it look like flakiness rather than a bug:

The failure count varied. Sometimes 1, sometimes 2, occasionally 0. A wrong-but-consistent answer reads as a bug; a varying answer reads as a race, and "it's flaky on CI" is where investigations go to die.

It only failed on the slower machine. Which inverts the usual intuition — a slower machine should give background work more time, not less.

Then a second symptom in a different test, same root cause:

@Test
fun timesOutAfterThirtySeconds() = runTest {
    val start = testScheduler.currentTime
    SyncManager(api, StandardTestDispatcher(testScheduler)).sync()
    assertEquals(96_000, testScheduler.currentTime - start)   // 3×30s + 2s + 4s
}

That one asserts the right thing and proves less than it appears to: it verifies the declared durations compose correctly, not that a real 30-second timeout ever fires.

Why the obvious fix failed#

"Add a delay at the end of the test to let things finish." The reflex, and it cannot work — delay inside runTest is virtual, so it returns immediately and waits for nothing. This is the single most instructive failure here: the tool you would reach for to wait is the tool that does not wait.

"Use Thread.sleep instead." It does block a real thread, and it is unavailable in commonMain/commonTest (there is no Thread on native or wasm), so the shared test cannot use it. On the JVM-only path it "fixes" the test by making it slow and still racy.

"Retry the test on CI." We did, for two weeks. It converted a real bug into a longer build, and the underlying defect — a production class that hard-codes its dispatcher — stayed.

"Increase the CI timeout." Addresses a symptom that was never the problem. The test was not timing out; it was asserting too early.

The actual mechanism#

runTest installs a TestScheduler and a TestDispatcher as the test coroutine's context. The scheduler holds a virtual current time.

delay consults the scheduler, not the OS. A delaying coroutine registers a resumption at virtual time T and suspends. When nothing else can run, the scheduler advances its clock to the next registered time and resumes it. No thread ever sleeps.

advanceUntilIdle() drains the scheduler's queue. It runs tasks and advances virtual time until the scheduler has nothing left. That is a statement about tasks the scheduler knows about.

A real dispatcher is not on that scheduler. Dispatchers.Default runs on a real thread pool with real timing. The test scheduler has no visibility into it, so advanceUntilIdle considers itself idle while that work continues.

Which makes the defect a property of the code under test, not of the test. A single withContext is enough to leave the scheduler's world:

// production code — the test can do nothing about this from outside
suspend fun sync() = withContext(Dispatchers.IO) {
    delay(30.seconds)          // a REAL 30 seconds; the virtual clock is not consulted
    api.push(pending)
}

No amount of advanceUntilIdle() reaches that delay, because the coroutine running it is not on the test scheduler. The test either waits thirty real seconds or — far more commonly — asserts while the work is still in flight and passes for the wrong reason.

So "use virtual time" is really "inject the dispatcher", and the two are the same requirement:

class Syncer(private val io: CoroutineDispatcher) {
    suspend fun sync() = withContext(io) { delay(30.seconds); api.push(pending) }
}

@Test fun syncRetriesAfterBackoff() = runTest {
    // StandardTestDispatcher shares THIS test's scheduler — so the delay is virtual
    val syncer = Syncer(StandardTestDispatcher(testScheduler))
    …
}

Passing testScheduler is the load-bearing part. A bare StandardTestDispatcher() constructs its own scheduler, which puts the work back outside the one advanceUntilIdle drains — the same bug, now wearing a test-dispatcher costume and considerably harder to spot in review.

So the race is between the test body's assertions and a thread pool the test cannot see — and a slower machine makes the pool later, which is why the slow machine failed more. The local machine was fast enough to finish the real work before the assertion by luck.

The rule:

Virtual time only governs coroutines on the test dispatcher. Anything that hops to a real dispatcher leaves the clock, and advanceUntilIdle will cheerfully report idle while it runs.

The fix#

Inject the dispatcher so the test can supply the test one. That is the whole fix, and it is the same injectability lesson that applies to clocks.

// commonTest
@Test
fun retriesThreeTimesThenFails() = runTest {
    val api = HangingApi()
    // StandardTestDispatcher shares THIS test's scheduler, so the work is on the virtual clock.
    val manager = SyncManager(api, StandardTestDispatcher(testScheduler))

    val result = manager.sync()

    assertEquals(SyncResult.Failed, result)
    assertEquals(3, api.attempts)
    assertEquals(96_000, currentTime)      // 3×30s timeout + 2s + 4s backoff
}

The production class already took a dispatcher — it was the test that passed a real one. Where it does not, that is the change to make:

// commonMain — dispatcher is a dependency, with a platform default
class SyncManager(
    private val api: Api,
    private val dispatcher: CoroutineDispatcher = platformIoDispatcher(),
)

Two further pieces are worth having.

A leak detector, so this class of bug fails loudly rather than flakily. runTest already fails a test whose child coroutines have not completed — the gap is work on a foreign dispatcher. An explicit assertion closes it:

// commonTest
@Test
fun noWorkEscapesTheTestDispatcher() = runTest {
    val api = CountingApi()
    SyncManager(api, StandardTestDispatcher(testScheduler)).sync()
    advanceUntilIdle()
    assertEquals(api.started, api.finished)   // nothing still in flight
}

And one real-time test, for the thing virtual time cannot prove. Keep exactly one, and name it so nobody "optimises" it away:

// commonTest — deliberately NOT runTest. Asserts a real wall-clock bound.
@Test
fun realTimeoutActuallyFires() = runBlocking {
    val elapsed = measureTime {
        withTimeoutOrNull(300.milliseconds) { HangingApi().pull() }
    }
    assertTrue(elapsed < 1.seconds, "timeout did not fire in real time: $elapsed")
}

Short duration, real clock, one assertion: that the timeout mechanism works at all. The virtual tests then cover the composition of durations, which is what they are good at. Using virtual time for the arithmetic and one real test for the mechanism is the division that actually holds.

How we proved it#

The bug is a race, so the honest proof is to make it deterministic rather than to run it until it passes.

Forcing the failure. A deliberate delay on the fake API's real-dispatcher path turns the one-in-six into every time:

with Dispatchers.Default + HangingApi(startupDelay = 50ms)
  run 1  FAILED  expected 3 attempts but was 0
  run 2  FAILED  expected 3 attempts but was 0
  run 3  FAILED  expected 3 attempts but was 0

Reproducing a flaky failure deterministically is the step that distinguishes a diagnosis from a guess. Before this, "slow machine" was a theory; after it, the mechanism is demonstrated.

After the fix, across the matrix:

jvmTest                retriesThreeTimesThenFails  PASSED  (12ms, currentTime=96000)
androidUnitTest        retriesThreeTimesThenFails  PASSED  (14ms, currentTime=96000)
iosSimulatorArm64Test  retriesThreeTimesThenFails  PASSED  (31ms, currentTime=96000)
wasmJsTest             retriesThreeTimesThenFails  PASSED  (48ms, currentTime=96000)

Identical currentTime on four targets is the useful signal: the assertion is now on virtual time, which is machine-independent by construction.

And 500 CI runs, because the original claim was about frequency:

before:  500 runs → 83 failures  (16.6%)
after:   500 runs →  0 failures

One honest limit, and it is the point of the real-time test above: nothing here proves that withTimeout(30.seconds) fires after thirty seconds on a real device. The virtual tests prove the retry logic composes the durations I declared. If the production dispatcher is misconfigured in a way that breaks real delays, this suite stays green — which is a gap I chose to cover with one cheap real-clock test rather than pretend away.

What this generalizes to#

A fake clock tests logic, not timing. This is true of every injected clock, every frozen time in a test framework, every simulated scheduler. You get determinism and speed; you give up evidence that the real mechanism fires. Keep one real-time test for the mechanism and use the fake for everything about the arithmetic.

"Wait for idle" is always scoped to what the waiter can see. advanceUntilIdle, an IdlingResource, await all tasks, a framework's flush() — each drains a queue it owns. Work that escaped to another executor is outside the promise, and the symptom is an assertion that reads too early.

Varying failure counts point at an unobserved concurrent actor. A consistently wrong value is a logic bug; a value that changes between runs means something you are not controlling is racing you. That reframing is what turns "flaky" into a findable bug.

Reproduce the flake deterministically before fixing it. Adding the 50ms startup delay took two minutes and converted a statistical argument into a demonstration. Retry-until-green is the alternative, and it is how a real defect survives for two weeks.

Related#

• A Flow is cold, and that is why your ViewModel fires three network calls — tests that count executions
• commonTest runs on every target, which is the point and the cost — why identical currentTime across targets matters
• kotlinx-datetime has no Calendar, and the timezone database ships per platform — the other injectable clock

Keep reading#

This is part of 100 Days of Kotlin Multiplatform — the boundaries, the sharp edges, and the things that only show up on the second target. Follow MobileByteSensei for the rest.