The rollout with no off switch, discovered at 2am

TL;DR#

A feature flag is only a kill switch if turning it off restores the previous behaviour. Ours gated which code path ran and not what that path had already written — so flipping it off left the system reading old code against new data.

The off position had never been exercised with production-shaped data. At 2am it turned out to be a third state: not the new behaviour, not the old one, but a combination that neither code path was written to handle.

Test the off switch before you need it, with data the on position has already touched. A flag whose off path is untested is a flag you do not have.

The case this post is about#

A migration to a new notification-preferences model, behind a flag:

// the new model writes a normalised table; the old one used a JSON blob
suspend fun preferences(userId: Long): Preferences =
    if (flags.isEnabled("prefs_v2")) prefsV2Repo.load(userId)
    else legacyPrefsRepo.load(userId)

suspend fun save(userId: Long, prefs: Preferences) {
    if (flags.isEnabled("prefs_v2")) prefsV2Repo.save(userId, prefs)      // writes prefs_rows
    else legacyPrefsRepo.save(userId, prefs)                              // writes users.prefs_json
}

Flag at 25%, then 50%, then 100% over four days. Monitored, canaried, and reviewed. The rollout itself went fine.

The symptom#

Night six, 02:14. An unrelated incident — a slow query on prefs_rows — and the on-call engineer did the right thing:

02:14  prefs_rows query p99 → 8s, notification sends backing up
02:16  on-call: disable prefs_v2      ← the documented mitigation
02:16  flag off, all instances
02:17  notification sends resume
02:19  support: "I'm getting emails I turned off three days ago"

Flipping the flag off restored the code path and the legacy code read users.prefs_json, which had not been written to since that user's preferences were last saved under v2. Three days of preference changes existed only in prefs_rows.

user 88214
  users.prefs_json   {"email_marketing": true,  "updated": "2026-10-28"}   ← stale
  prefs_rows         email_marketing = false,   updated 2026-11-01         ← current, ignored

The blast radius:

users who changed preferences while v2 was on:       14,208
of those, who had opted OUT of something:             3,916
marketing emails sent to opted-out users before
the flag was turned back on (41 minutes):            2,841

Two thousand eight hundred and forty-one emails to people who had unsubscribed. A compliance incident, not a performance one.

Why the obvious fix failed#

"Turn the flag back on." Which is what we did at 02:55, and it stopped the sends — while leaving the original slow-query incident unmitigated. The on-call engineer was left with two bad options and no good one, which is the actual failure.

"Document that the flag is one-way." We did, afterwards, and a documented one-way flag is a rollout with no mitigation. The runbook now said "do not disable this", which means the incident response for any v2 problem is "there isn't one".

"Backfill prefs_json from prefs_rows on demand." The eventual fix for the data, and it took 40 minutes to write under pressure at 3am — exactly the thing you do not want to be authoring during an incident.

"Test the flag off in staging." It was tested, and staging's data had only ever been written by whichever path was active at the time. The off position was tested against clean data, which is not the state that matters.

The actual mechanism#

A flag gates behaviour. A migration changes state. Those have different reversibility properties, and a single flag over both is the bug.

flag ON   → new code path → writes new state
flag OFF  → old code path → reads OLD state

the third state nobody designed:
flag OFF  → old code path → reads old state, which is now STALE
            because the new path wrote elsewhere

The asymmetry is what makes this specifically a trap. Turning the flag on is reversible for as long as nothing has been written — which is the window in which you test the rollback, find it works, and conclude the flag is safe. Every write after that point moves state out from under the old path, so the flag's reversibility decays with traffic rather than being a fixed property.

That decay has no alarm and no representation anywhere. A flag at 1% for an hour is genuinely reversible; the same flag at 50% for a day is not, and nothing in the flag UI distinguishes them. Both show a toggle in the same position, which is why "we can always turn it off" survives as a belief long after it has stopped being true.

The useful reframing: a flag over a state change is not an off switch, it is a point of no return with a delay. The only question worth asking before enabling it is not "can we turn this off" but "what reconciles the state we are about to diverge if we do".

Three properties made this specifically dangerous:

The off position was never exercised against dirtied data. Staging data was written by one path at a time. The failure requires data written by the new path and read by the old one, which is a state that only exists after a rollout.

It failed silently and plausibly. Reading stale preferences is not an error. The legacy path worked perfectly and returned an answer that was three days out of date, with nothing to signal it.

And the flag flip was fast and global. Twenty seconds from decision to 100% of instances, which is the right property for a kill switch and the wrong one for something whose off position is untested.

A flag is a kill switch only if off restores the prior behaviour. When the on path writes state the off path cannot see, the flag is a one-way door with a two-way interface — which is worse than a one-way door, because someone will open it.

The fix#

Dual-write during the rollout, so both stores stay current and the off position is genuinely correct.

Phase 1 — dual-write, read from the old store:

suspend fun save(userId: Long, prefs: Preferences) {
    legacyPrefsRepo.save(userId, prefs)                 // ALWAYS — keeps the off path valid
    if (flags.isEnabled("prefs_v2_write")) {
        runCatching { prefsV2Repo.save(userId, prefs) }  // additive; a failure must not block
            .onFailure { log.atWarn().addKeyValue("event", "v2_write_failed").log(it.message) }
    }
}

suspend fun preferences(userId: Long): Preferences =
    if (flags.isEnabled("prefs_v2_read")) prefsV2Repo.load(userId)
    else legacyPrefsRepo.load(userId)

Two flags, not one — the single most important change. A read flag and a write flag, flipped independently:

state                        read from   written to      off-switch safe?
v2_write=off v2_read=off     legacy      legacy          yes (baseline)
v2_write=on  v2_read=off     legacy      both            yes ← soak here
v2_write=on  v2_read=on      v2          both            yes ← and here
v2_write=on  v2_read=off     legacy      both            yes (the rollback)

Turning v2_read off is now safe at any point, because legacy has been kept current throughout.

Phase 2 — a divergence check, running continuously:

// a sampled comparison, so "both stores agree" is measured rather than assumed
@Scheduled(every = "1m")
suspend fun checkDivergence() {
    sampleUserIds(100).forEach { id ->
        val a = legacyPrefsRepo.load(id)
        val b = prefsV2Repo.load(id)
        if (a != b) metrics.counter("prefs_divergence").inc()
    }
}

With an alert on non-zero. A divergence means the off switch has stopped being safe, and that is precisely when you want to know.

Phase 3 — stop dual-writing only after the read has soaked, and keep a reverse backfill ready:

# written and TESTED during phase 2, not at 3am
$ ./scripts/backfill-prefs-json-from-rows.sh --since 2026-10-28 --dry-run

And the practice that would have caught all of it: a rollback drill, in staging, against data the on path has written:

1. enable v2_read + v2_write in staging
2. drive realistic traffic that CHANGES preferences for 1,000 users
3. disable v2_read
4. assert: preferences read == preferences last written

Step 2 is the one the original testing skipped.

How we proved it#

That the original flag was not reversible, reproduced in staging with the drill above:

1,000 users, preferences changed under v2, then v2_read disabled
  users whose preferences read back correctly:      0 / 1,000
  users whose read-back was stale:                1,000

Zero of a thousand. That result was available for the cost of a thirty-minute drill, before the rollout.

That dual-write makes the off position correct:

same drill, with dual-write enabled
  users whose preferences read back correctly:   1,000 / 1,000
  divergence counter during the run:                 0

And the divergence monitor catching a real problem during the actual re-rollout — which is the result that justified building it:

day 2 of the re-rollout
  prefs_divergence: 0 … 0 … 14   ← alert
  cause: a batch importer wrote legacy directly, bypassing the dual-write path
  → importer updated; divergence back to 0 within 20 minutes

A fourth write path nobody had enumerated. Without the monitor, the off switch would have been silently unsafe for that cohort, and we would have found out the next time someone flipped it.

The remediation of the original incident, stated plainly because it is the cost:

opted-out users emailed:            2,841
users notified of the error:        2,841  (apology + confirmation of their current setting)
regulatory notification:            filed (unsubscribe honoured late)
preferences restored from prefs_rows: 14,208 users, backfill completed 04:40

One limit worth naming: dual-write makes the write path slower and introduces a window where one store has a value the other does not — ours is sub-second and bounded by the runCatching ordering (legacy first, so a v2 failure never loses data), but it is a real inconsistency window rather than an atomic write. For preferences that is acceptable. For money it would not be, and the answer there is a transactional outbox rather than best-effort dual-write.

What this generalizes to#

A reversible flag and an irreversible migration cannot share one switch. Separate the flags for reading and writing, and the state transitions become enumerable — each with an answer to "is off safe here?".

Test the off position with data the on position produced. The failure state requires the system to have run in the new mode. Clean-slate testing of a rollback is testing the wrong scenario, and it is the default.

A silent wrong answer is the worst rollback failure. Stale preferences read as valid data. If the off path can return something plausible-and-wrong, add a staleness check rather than trusting the read.

Write the recovery script before you need it. The backfill took 40 minutes to author at 3am under incident pressure. The same script written during phase 2, with a --dry-run and a test, is the difference between a 41-minute incident and a 5-minute one.

Related#

• The config change that was really a deploy, and had none of the safeguards — fast global changes
• The migration that could not be rolled back — the same dual-write discipline for schema
• A feature flag read on four platforms needs one source of truth and four caches — flags as state

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.