iosTest needs a booted simulator, and CI does not have one by default

TL;DR#

jvmTest runs in the Gradle daemon's JVM. iosSimulatorArm64Test compiles a native executable and runs it on a booted simulator through xcrun simctl. If no simulator matching the configured device is booted, the task fails — and the message talks about the toolchain rather than about a missing runtime.

So the shared tests you wrote to get multi-target coverage silently do not run on iOS in CI, which is the one target they were most needed for. The task is often skipped rather than failed, so the build is green.

Pin the device name, boot it explicitly, and assert that the iOS test task actually ran rather than trusting a green build.

The case this post is about#

A shared module with real commonTest coverage, and CI configured the obvious way:

# .github/workflows/ci.yml
- run: ./gradlew allTests

// build.gradle.kts
kotlin {
    iosArm64()
    iosSimulatorArm64()
    jvm("desktop")
    androidTarget()
}

Green builds for weeks. Then a Float.toString() difference reached production on iOS — the exact class of bug commonTest on multiple targets exists to catch, from a test suite that had been green the whole time.

The symptom#

Three findings, and the first one is the reason this went unnoticed.

The iOS test task was being skipped, not failed. In the build output, between hundreds of lines:

> Task :shared:iosSimulatorArm64Test SKIPPED
> Task :shared:iosArm64Test SKIPPED

allTests completed, exit code 0, and two of four targets never executed a single test. A skipped task is not a failure, and nothing summarised the difference.

When it did run, the failure named the wrong thing. On a machine with Xcode but no booted device:

> Task :shared:iosSimulatorArm64Test FAILED
  Could not find a simulator with name 'iPhone 15'
  …
  Execution failed: xcrun simctl spawn … exited with code 164

Exit code 164 and an xcrun invocation reads as a broken Xcode install. Two engineers reinstalled command-line tools before anyone checked simctl list.

And iosArm64Test cannot run at all. It targets a physical device. On CI there is no device, so the task has nothing to run on — which is correct behaviour and looks identical to the simulator problem in the log.

Why the obvious fix failed#

"allTests runs all the tests." It runs all the test tasks that are runnable. A task with no available device is skipped, and allTests reports success. The name is doing a lot of work here.

"Install Xcode on the runner." Already installed — macos-14 runners ship it. The missing piece was a booted simulator, which is a separate thing from the toolchain.

"Reinstall the command-line tools." The 164 exit code and xcrun in the message make this irresistible. It changed nothing, twice.

"Add --continue so a failure does not stop the build." Makes it worse: now genuine iOS test failures also do not fail CI.

"Skip iOS tests in CI and run them locally." What we did for a week. Local runs happen when someone remembers, which is never for the target nobody develops on.

The actual mechanism#

A Kotlin/Native test task is a three-step pipeline, and only the first step resembles a JVM test run.

Compile a test executable. linkDebugTestIosSimulatorArm64 produces a real native binary containing your tests and a test runner.

Find a device. The task resolves a simulator by name — from kotlin.native.cocoapods settings, or the Kotlin plugin's default, or an explicit device configuration. The name must match a device in simctl list exactly.

Spawn the binary on it. xcrun simctl spawn <udid> <binary> runs the executable inside the simulator runtime and parses its output.

Steps two and three are why the failure modes look odd:

No matching device → the task cannot run. Depending on configuration, that surfaces as a SKIPPED task or a failure referencing xcrun. Neither says "boot a simulator".

iosArm64Test targets hardware, so it has no simulator to fall back to and is skipped unconditionally in CI. That is correct and should be stated rather than discovered.

A native test task needs a runtime host, and a missing host is not a compile error. The default is to skip, so the absence of coverage is reported as success.

The sub-case that brings the bug back months after you fix it: the device name is matched exactly, and Xcode changes the device list. Pin "iPhone 15", take an Xcode update that ships the iOS 18.2 runtime, and the installed set becomes:

$ xcrun simctl list devices available | grep iPhone
    iPhone 16 (B2C1…) (Shutdown)
    iPhone 16 Pro (9F0A…) (Shutdown)

No iPhone 15. The build does not fail — step two finds no match, and you are back to a skipped task and a green tick, with the added cruelty that the configuration was correct when it was written and nobody changed it.

The defence is not a better name. It is to resolve the device at runtime and fail when the resolution comes back empty:

UDID=$(xcrun simctl list devices available -j \
  | jq -r '[.devices[][] | select(.name|startswith("iPhone"))][0].udid')
[ -n "$UDID" ] || { echo "no iPhone simulator available"; exit 1; }

Which is the same lesson as the main fix, applied to the thing doing the fixing: any step that can resolve to "nothing to do" has to distinguish that from "done", or it will eventually report the first as the second.

The fix#

Pin the device, boot it, and assert the tasks ran.

Pin the simulator name in Gradle, so it is not an environment accident:

// build.gradle.kts
kotlin {
    iosSimulatorArm64 {
        testRuns.configureEach {
            // Must match `xcrun simctl list devices` exactly.
            deviceId = providers.gradleProperty("iosTestDevice").getOrElse("iPhone 16")
        }
    }
}

Boot it in CI before the test task, and wait for it to actually be ready:

- name: Boot a simulator for native tests
  run: |
    set -euo pipefail
    DEVICE="iPhone 16"
    UDID=$(xcrun simctl list devices available -j \
           | python3 -c "import json,sys;d=json.load(sys.stdin)['devices'];
print([x['udid'] for v in d.values() for x in v if x['name']=='$DEVICE'][0])")
    xcrun simctl boot "$UDID" || true          # already-booted is fine
    xcrun simctl bootstatus "$UDID" -b         # BLOCK until the runtime is ready
    xcrun simctl list devices booted

bootstatus -b is the line that matters. simctl boot returns immediately and the runtime takes several seconds; without the wait, the test task races the boot and fails intermittently — which is how a fixed CI job becomes a flaky one.

Then assert the tasks ran, rather than trusting the exit code. This is the actual fix for the original bug:

- name: Run tests and verify every target executed
  run: |
    ./gradlew allTests --continue | tee build-log.txt || true
    for t in iosSimulatorArm64Test jvmTest testDebugUnitTest; do
      if grep -qE "Task :shared:$t (SKIPPED|NO-SOURCE)" build-log.txt; then
        echo "❌ $t did not run"; exit 1
      fi
    done

A test suite that did not execute is now a red build. That single check is worth more than the simulator configuration, because it catches every future variant of "the tests quietly stopped running".

And be explicit about iosArm64Test, which genuinely cannot run without hardware:

// build.gradle.kts — the device target compiles; it does not test on CI.
// Simulator coverage is the CI contract; iosArm64 is covered by compilation only.

Documenting that distinction stops the next person from "fixing" a skip that is correct.

A better per-target summary also helps, because the default output buries this:

tasks.withType<AbstractTestTask>().configureEach {
    testLogging { events("skipped", "failed") }
    afterSuite(KotlinClosure2({ desc: TestDescriptor, r: TestResult ->
        if (desc.parent == null) println("${project.name}:${name} → ${r.testCount} tests, ${r.resultType}")
    }))
}

How we proved it#

That tests were not running before. Reading the task outcome rather than the exit code:

$ ./gradlew allTests | grep -E 'Test (SKIPPED|FAILED)|Test$'
> Task :shared:iosSimulatorArm64Test SKIPPED
> Task :shared:iosArm64Test SKIPPED
$ echo $?
0

Exit code 0 with two targets skipped is the whole bug, in four lines.

That they run now, with a count:

$ ./gradlew allTests
shared:jvmTest                → 48 tests, SUCCESS
shared:testDebugUnitTest      → 48 tests, SUCCESS
shared:iosSimulatorArm64Test  → 48 tests, SUCCESS

Three suites of the same 48 shared tests. Before the fix, that third line did not exist.

That the guard catches a regression. Verified by removing the boot step:

$ (no simulator booted) ./gradlew allTests --continue | tee build-log.txt
> Task :shared:iosSimulatorArm64Test SKIPPED
$ ./check-tasks-ran.sh
❌ iosSimulatorArm64Test did not run
exit 1

And that the boot wait removes the flakiness — because the first version of this fix was itself intermittent:

without bootstatus -b:  20 CI runs → 6 failures ("Could not find a booted simulator")
with bootstatus -b:     20 CI runs → 0 failures

That table is the reason the wait is in the snippet. A fix that works most of the time reintroduces exactly the trust problem it was meant to solve.

One limit: this gives simulator coverage, not device coverage. iosArm64Test still does not run in CI, so anything that differs between simulator and hardware — and some things do — is covered by compilation only. That is a stated gap rather than a closed one.

What this generalizes to#

A skipped check reports as success. This is the most dangerous default in any test system. Whenever coverage depends on an available resource — a device, a database, a browser, a GPU — the absence of that resource must fail, not skip. Assert that the check ran.

"Run all tests" commands run all runnable tests. The aggregate task is a convenience, not a contract. If you depend on specific coverage, name the tasks and verify them individually.

An error message naming the plumbing will send you to the plumbing. xcrun and exit code 164 cost two reinstalls. When a message points at a tool, check the tool's own view of the world — simctl list — before rebuilding it.

A fix that is intermittently correct is not a fix. Booting without waiting looked like it worked and failed 30% of the time, which would have re-taught the team to ignore iOS test failures. Measure a fix across enough runs to see its tail.

Related#

• commonTest runs on every target, which is the point and the cost — the coverage this was silently losing
• One Gradle build, five outputs, and the cache that only helps three — compiling every target affordably
• The desktop target is the cheapest bug oracle you are not using — the fast local complement

Keep reading#

This is part of 100 Days of Kotlin Multiplatform — the boundaries, the sharp edges, and the things that only show up on the second target. Follow MobileByteSensei for the rest.