A staged rollout you control on one store and not on the others
Four channels, four halt latencies, and only one of them is under your hand.
TL;DR
- A bad release reached 100% of web users in 40 seconds and 18% of Android users in the 42 minutes before it was halted, while sitting in App Store review for 31 hours so no iOS user ever saw it. Halting it took four different actions with four different latencies.
- The store controls are not a rollout mechanism you can coordinate. They are four independent distribution systems whose halt times differ by three orders of magnitude.
- So the kill switch cannot live in a store. It has to live in the app, behind a remote flag the app reads at launch — and the defaults have to be per platform, which is its own trap.
- What reading on buys: the measured halt latency per channel, the flag shape that works across all four, and the drill that proves it.
The case this post is about
One release, version 4.12.0, shipped from one commit to four channels on the same afternoon:
15:02 web wrangler deploy → live
15:04 android Play Console, staged rollout 20% → rolling
15:06 ios App Store Connect, phased release, submitted → in review
15:08 desktop GitHub release + Sparkle appcast update → available
The defect: a migration in the shared module moved the notification preference to a new key name, and the read path looked only at the new key — with nothing migrating the value already stored under the old one. Every existing user's setting read as absent and fell through to the default. Not a crash — a silent reset, which meant no crash report and no alert.
It was found at 15:41 by someone on the team noticing their own settings had reverted.
The symptom
The halt took 94 minutes, and most of that was waiting rather than acting:
15:41 defect identified
15:43 web previous deployment promoted → 40s to propagate
15:46 android staged rollout HALTED in Play Console → stops NEW installs only
15:52 ios phased release PAUSED → still in review, no users yet
15:58 desktop appcast reverted to 4.11.3 → clients poll every 24h
17:15 android 4.12.1 submitted with the fix
The individual actions took seconds. What the table hides is the part that mattered: halting the rollout did nothing for the users who already had it.
users on 4.12.0 at 15:46, by platform
web 100% (everyone — one deploy, no gradual anything)
android 18% (41,208 devices, already installed — of a 20% stage)
ios 0% (still in review)
desktop 3% (whoever polled in that window)
Rolling back the web deployment fixed 100% of web users in 40 seconds. Halting the Android rollout fixed zero of the 41,000 devices that already had the build — it only stopped the number growing. Those users were on the broken version until they took an update, which for most of them was days.
Why the obvious fix failed
The first plan was to use each store's rollback. Android has one:
Play Console ▸ Release ▸ Halt rollout
Which stops distribution and does not uninstall or downgrade anything. There is no mechanism that removes a shipped APK from a device, by design. The 41,000 stayed broken.
The second plan was to ship 4.12.1 fast, which we did, and the timeline is the argument against relying on it:
17:15 4.12.1 submitted to Play, staged at 100%
17:52 available
→ 50% of affected devices updated within 19 hours
→ 90% within 4 days
→ the long tail never, because some devices have auto-update off
A forward fix is the right action and it is not a halt. Four days to 90% is the floor for a store-distributed fix, and that floor is set by user behaviour rather than by anything we control.
The third plan — and this is the one that took a drill to disprove — was to make the rollouts slower and more aligned, so that a bad release would reach fewer people before detection. That helps Android and desktop. It does nothing for web, where the deploy is atomic, and nothing for the already-installed cohort on any platform. It optimises the wrong variable: not how many people get it but how fast we can take it back.
The actual mechanism
Four channels, four distribution models, and the halt latency differs by three orders of magnitude:
Web is atomic and instantly reversible. One artifact, served from an edge; a deploy is a pointer change and so is a rollback. 40 seconds, 100% of users, in both directions. The best halt latency of the four, and it is the only channel where the store control is also the kill switch.
Android staged rollout controls distribution, not installations. Halting stops new installs. Installed builds persist. The only downgrade path is a user manually installing an older APK, which for a Play-distributed app is not a path.
iOS phased release is slower and the same shape. Seven days, 1% → 100%, pausable — and
Remove from sale stops new downloads without touching installed apps. There is also review
latency in front of it, which on this occasion helped us and is not a control.
Desktop is a polling client. Reverting the appcast means clients that poll after the revert see the old version. Clients that polled before it already have the new one, and the poll interval is in the client — which is in the build you are trying to replace.
So three of the four channels have the same property: you can stop the spread and you cannot recall the release. The thing that can be recalled is not the binary — it is the behaviour, and only if the binary was built to ask.
That reframes the whole problem. A staged rollout is a blast-radius control, which is worth having and is not a rollback. The rollback has to be something the already-installed app consults, which means it has to be in the app, which means it has to have been there before the release you want to halt.
The fix
A remote kill switch the app reads at launch and on resume, with the new feature behind it, and per-platform defaults that fail safe.
// commonMain/kotlin/release/KillSwitch.kt
//
// The one control that works on an INSTALLED build. Read at launch and on resume; the
// features it gates must tolerate being turned off at any moment, which is a design
// constraint on the feature rather than on this file.
class KillSwitch(
private val remote: RemoteConfig,
private val local: Prefs,
private val platform: Platform,
) {
/**
* The default when the network is unavailable, and the subtle part of the whole design.
* It is PER PLATFORM because the halt latency is: a web user can be fixed by a deploy in
* 40 seconds, so defaulting a web client to "on" risks 40 seconds of exposure. An
* installed Android build may be days from a fix, so it defaults to the LAST KNOWN
* value — which, once a kill has been seen, stays killed across restarts.
*/
private fun fallback(key: String): Boolean = when (platform) {
Platform.Web -> true // a fix is seconds away
else -> local.getBoolean(key, default = true) // sticky: a seen kill persists
}
suspend fun isEnabled(key: String): Boolean {
val fetched = remote.boolOrNull(key) ?: return fallback(key)
// Persist EVERY fetch, not just the kills. Persisting only `false` would mean a
// feature re-enabled upstream stays dead on a device that is then offline — a
// one-way switch, which is a different outage.
local.putBoolean(key, fetched)
return fetched
}
}
The migration that caused this, rewritten to be haltable — which is the part that is a design constraint rather than a config change:
// commonMain/kotlin/storage/NotificationPrefs.kt
//
// Reads BOTH keys regardless of the switch, and writes the new one only when enabled. That
// asymmetry is what makes the kill switch meaningful: turning it off stops new writes and
// every already-migrated device still reads correctly, so the halt does not create a second
// cohort of broken users.
class NotificationPrefs(private val prefs: Prefs, private val killSwitch: KillSwitch) {
suspend fun enabled(): Boolean =
prefs.getBooleanOrNull(KEY_NEW) // migrated devices
?: prefs.getBooleanOrNull(KEY_OLD) // not yet migrated — the bug was omitting this
?: DEFAULT
suspend fun setEnabled(value: Boolean) {
if (killSwitch.isEnabled("prefs_migration_v2")) {
prefs.putBoolean(KEY_NEW, value)
prefs.putBoolean(KEY_OLD, value) // dual-write while the migration is live
} else {
prefs.putBoolean(KEY_OLD, value) // halted: old key only
}
}
private companion object {
const val KEY_NEW = "notifications.enabled"
const val KEY_OLD = "notify_on"
const val DEFAULT = true
}
}
And the output it produced, on the drill described below:
$ ./tools/killswitch-drill.sh --flag prefs_migration_v2 --platforms all
flag flipped at 14:02:11
web 14:02:14 (+3s) 1,000 / 1,000 clients observed the kill
android 14:02:49 (+38s) 1,000 / 1,000
ios 14:03:02 (+51s) 1,000 / 1,000
desktop 14:04:36 (+145s) 1,000 / 1,000
read-back: 1,000 / 1,000 devices report the OLD key value correctly
Under three minutes on every channel, against 94 minutes and 41,000 unfixed devices.
How we proved it
That halting a rollout does not fix installed users — the number that reframed the incident:
Play Console ▸ Statistics, 4.12.0, at the moment of halt
installs 41,208
halted yes
installs 1h later 41,208 ← spread stopped
affected users fixed by the halt 0
That the kill switch reaches installed builds, measured per channel rather than assumed. The drill flips a flag against 1,000 synthetic clients per platform and records when each observes it:
flip → observed (p99) flip → BEHAVIOUR changed
web 3s 3s
android 38s 38s
ios 51s 52s
desktop 145s 147s
The second column is the one that matters and is the one a config dashboard does not show. A client that has fetched the flag and not yet acted on it is not halted, and the two numbers being nearly equal is the evidence that the feature actually re-reads rather than caching its decision at startup.
That the sticky fallback does not become a one-way switch. This is the branch I got wrong first, and the fixture is the sequence that exposes it:
$ ./gradlew :release:test --tests '*ReEnableWhileOffline*'
1. flag = false, fetched, persisted → feature off
2. flag = true upstream, device OFFLINE → feature off (correct: no fetch)
3. device online, fetch succeeds → feature ON ← failed before the fix
The original persisted only false, so step 3 left the feature dead forever on any device that
had ever seen a kill. That is an outage with no alert, and it would have been found by a user
rather than by us.
That the dual-write makes the halt safe for already-migrated devices. A fixture with a device that migrated, then had the flag killed:
$ ./gradlew :release:test --tests '*HaltAfterMigration*'
device migrated (new key written), flag killed, setting changed
expected old-key value: false
actual: false ← the dual-write kept it in sync
Without the dual-write, halting the migration would have created a second broken cohort — devices whose truth lived in the new key while the code had gone back to reading the old one. That is the mistake that turns a rollback into a second incident.
What I cannot tell you is the halt latency on a cold-start-only read. Our features re-read on resume, which is why the two drill columns agree; an app that reads config once at process start would have a halt latency equal to its session length, and we have not measured that distribution. If your features cache their flag decisions, the numbers above do not apply to you.
And an honest limit of the whole approach. A kill switch cannot halt a defect in the code that reads the kill switch, or in anything that runs before it — a crash in the app's initialisation path is not haltable by any mechanism in the app. For that class, the store controls and the forward fix are all there is, which is the argument for keeping the staged rollout as well rather than instead.
What this generalizes to
A distribution control and a rollback are different things, and platforms with the best-looking rollout tooling often have the worst recall. The question to ask of any release channel is not how gradually can I ship this but how fast can I take it back from someone who already has it — and on three of four channels here the answer is "you cannot". Once that is the answer, the reversibility has to be built into the artifact before it ships, which makes it a design requirement on the feature rather than an operational capability you can add later.
Related
- A flag over a state change is not an off switch — the dual-write in this post, and what happens when you skip it.
- One version, four numbering schemes — the other half of releasing to four channels, and the hotfix that could not be numbered.
Keep reading
One post a week on shipping mobile software. No digests, no course pitch. Subscribe.
One post a week, no digest
New writing when it lands. No roundups, no course pitch, no "10 tips".