Remote config defaults that disagreed, and the feature that shipped twice

TL;DR#

• One remote-config value, four platform SDKs, four locally-declared defaults — and they disagreed. On a slow first launch the same correct configuration produced a free tier on iOS and a paid tier on Android, for 1,340 users over nine days.
• The fetched value is shared. The fallback is whatever each SDK's local default map says, declared in platform code, and nothing compared the four.
• The fix makes the defaults a single shared artifact that every actual reads, and makes the source of each value observable so a default-sourced read is visible rather than assumed to be a fetch.
• What reading on buys: the four defaults measured, the first-launch window that exposes them, and the test that fails when they drift.

The case this post is about#

A paywall gate, configured remotely:

// commonMain/kotlin/config/RemoteConfig.kt
expect class RemoteConfig {
    suspend fun fetch(): Boolean
    fun boolean(key: String): Boolean
    fun long(key: String): Long
}

// androidMain — Firebase Remote Config
actual class RemoteConfig(private val fb: FirebaseRemoteConfig) {
    init {
        fb.setDefaultsAsync(mapOf(
            "paywall_enabled" to true,         // ← declared here
            "free_note_limit" to 10L,
        ))
    }
    actual fun boolean(key: String) = fb.getBoolean(key)
    …
}

// iosMain — the same SDK through Obj-C, different defaults
actual class RemoteConfig(private val rc: RemoteConfig) {
    init {
        rc.setDefaults(mapOf(
            "paywall_enabled" to false,        // ← and here, differently
            "free_note_limit" to 25L,
        ) as Map<Any?, Any?>)
    }
    …
}

// desktopMain, wasmJsMain — hand-written, with a third and fourth set
private val defaults = mapOf("paywall_enabled" to false, "free_note_limit" to 50L)

Four maps, written at four different times by three different people, each one reasonable in isolation.

The symptom#

A support pattern that looked like a billing bug:

"My phone shows the paywall after 10 notes. My iPad lets me add as many as I want."

Same account, same remote config, different behaviour. And the remote config itself was correct and identical for both:

# Firebase console, the only source of truth anyone had checked
  paywall_enabled   true
  free_note_limit   10

The divergence was entirely in the window before the first successful fetch:

# measured: time from cold start to a completed fetch
                    p50       p90      p99      failed entirely
  android          310 ms    1.2 s    4.1 s         0.4%
  ios              290 ms    1.1 s    3.8 s         0.3%
  desktop          340 ms    1.4 s    5.2 s         1.1%
  web              180 ms    0.7 s    2.2 s         2.9%

# users who reached the note-limit check BEFORE the first fetch completed, 9 days
  1,340

Thirteen hundred users made a paywall decision from a local default, and the local defaults disagreed — so free_note_limit was 10 on Android and 25 on iOS for the same person in the same minute.

And a second, quieter consequence:

# users who NEVER completed a fetch in a session (offline, blocked, SDK failure)
  web      2.9%    → ran on free_note_limit = 50 for the whole session

Why the obvious fix failed#

The first fix was to make the Android default match iOS:

"paywall_enabled" to false,
"free_note_limit" to 25L,

Which aligns two of four and leaves the same class of bug between Android and desktop. It also picks the permissive value, so a fetch failure now gives away paid features rather than wrongly charging for them — which is the better direction and was not a decision anyone made deliberately.

The second fix was to block the UI until the fetch completes:

suspend fun start() {
    config.fetch()                  // await it before showing anything
    showHome()
}

Correct behaviour and a measurable cost: p99 is 4.1 seconds on Android and the fetch fails outright for 2.9% of web sessions, so that 2.9% would see a loading screen forever. We did ship a bounded version of this for the paywall check specifically, and not as a startup gate.

The third attempt was to put the defaults in a shared constant and pass them to each SDK:

// commonMain
val CONFIG_DEFAULTS = mapOf("paywall_enabled" to false, "free_note_limit" to 25L)

Which is most of the answer and was not sufficient, because the SDKs do not agree on types. Firebase on iOS, through Obj-C, stores a Long default as an NSNumber and getNumberValue returns a Double — so 25L round-tripped as 25.0 and a .toLong() on a value above 2^53 would have lost precision. Only one of our values is a Long and it is small, and the shape of that problem is why the fix below reads the source of a value rather than trusting it.

The actual mechanism#

A remote-config read has two possible sources and the API does not distinguish them.

A fetched value is shared. All four SDKs pull the same server-side configuration and, once fetched, agree. This part worked and is why the console looked correct.

A default is declared locally, per SDK, in platform code. It is the value returned before a fetch completes, and after a fetch that fails. There is no mechanism that compares the four declarations — they are four init blocks in four source sets, and nothing in commonMain can see any of them.

And boolean(key) returns a Boolean either way. The caller cannot tell whether it received a fetched value or a fallback, so it cannot know whether the decision it is about to make is based on the configuration or on a guess.

The first-launch window is where this is observable, and it is not small:

  cold start → fetch complete, p90      1.1–1.4 s
  cold start → the note-limit check     ~400 ms   (the home screen's first composition)

The check runs before the fetch completes for most cold starts, so the default is the normal path for a first launch rather than an edge case — which inverts the usual intuition that defaults are rarely used.

Plus a third source nobody had declared: a value present in neither the fetch nor the local defaults returns the SDK's own type-zero — false, 0L, "" — so a typo'd key reads as false with no error, and paywall_enabled misspelled is a silently disabled paywall.

The fix#

One shared defaults artifact that every actual consumes, a read that reports its source, and a bounded wait at the decision point rather than at startup.

// commonMain/kotlin/config/ConfigDefaults.kt
//
// ONE declaration, consumed by every actual. Four init blocks in four source sets cannot be
// compared by anything, which is how 10 and 25 coexisted for nine days.
//
// Every default is the SAFE-ON-FAILURE value, chosen deliberately: a fetch failure should give
// a user slightly too much rather than wrongly charge them. That is a product decision and it
// is recorded here rather than being a side effect of whoever wrote each map.
object ConfigDefaults {
    val values: Map<String, ConfigValue> = mapOf(
        "paywall_enabled" to ConfigValue.Bool(false),    // fail open: no paywall
        "free_note_limit" to ConfigValue.Int64(25),       // fail generous
        "sync_interval_min" to ConfigValue.Int64(15),
    )
    val keys: Set<String> = values.keys
}

sealed interface ConfigValue {
    data class Bool(val v: Boolean) : ConfigValue
    data class Int64(val v: Long) : ConfigValue
}

/** Where a read came from. The thing a bare Boolean return could not carry. */
enum class ConfigSource { Fetched, LocalDefault, Missing }

data class ConfigRead<T>(val value: T, val source: ConfigSource)

expect class RemoteConfig {
    suspend fun fetch(): Boolean

    /**
     * Returns the value AND its source. A caller making an irreversible or billing-relevant
     * decision needs to know whether it is acting on configuration or on a guess, and the
     * original Boolean return made those indistinguishable.
     */
    fun boolean(key: String): ConfigRead<Boolean>
    fun long(key: String): ConfigRead<Long>
}

// iosMain
actual class RemoteConfig(private val rc: FIRRemoteConfig) {
    init {
        // Built from the SHARED map. Nothing is declared here.
        rc.setDefaults(ConfigDefaults.values.mapValues { (_, v) ->
            when (v) { is ConfigValue.Bool -> v.v; is ConfigValue.Int64 -> v.v }
        } as Map<Any?, Any?>)
    }

    actual fun long(key: String): ConfigRead<Long> {
        // A key absent from the shared declaration returns Missing rather than the SDK's
        // type-zero. A typo'd `paywall_enabled` previously read as `false` — a silently
        // disabled paywall, with no error anywhere.
        if (key !in ConfigDefaults.keys) {
            logger.error("config: unknown key '$key' — not in ConfigDefaults")
            return ConfigRead(0L, ConfigSource.Missing)
        }
        val v = rc.configValueForKey(key)
        // The source is the SDK's own, so it is authoritative rather than inferred from
        // whether a fetch has completed — a fetch can succeed and still not contain a key.
        val source = when (v?.source()) {
            FIRRemoteConfigSourceRemote  -> ConfigSource.Fetched
            FIRRemoteConfigSourceDefault -> ConfigSource.LocalDefault
            else                         -> ConfigSource.Missing
        }
        // numberValue is a Double through the Obj-C binding; a Long above 2^53 would be
        // approximated, so the conversion is explicit and the ceiling is asserted.
        val n = v?.numberValue()?.doubleValue ?: 0.0
        require(n <= 9_007_199_254_740_991.0) { "config '$key' exceeds the Double-safe range" }
        return ConfigRead(n.toLong(), source)
    }
}

// commonMain — the decision point, with a bounded wait rather than a startup gate
class PaywallGate(private val config: RemoteConfig) {

    suspend fun shouldShowPaywall(noteCount: Int): Boolean {
        var limit = config.long("free_note_limit")

        // Wait only HERE, only briefly, and only if we would otherwise decide on a default.
        // A startup gate costs every launch p99 4.1 s; this costs 800 ms on the one screen
        // where being wrong is a billing error.
        if (limit.source != ConfigSource.Fetched) {
            withTimeoutOrNull(800.milliseconds) { config.fetch() }
            limit = config.long("free_note_limit")
        }

        // Still a default: proceed on the SAFE value and record it, so the rate of
        // default-sourced billing decisions is a number somebody can see.
        if (limit.source != ConfigSource.Fetched) {
            analytics.log("config.decided_on_default", "key" to "free_note_limit")
        }
        return config.boolean("paywall_enabled").value && noteCount >= limit.value
    }
}

And the test that fails when the four drift again — which they cannot now, but the test is what makes that true rather than currently-true:

// commonTest/kotlin/config/ConfigDefaultsTest.kt
class ConfigDefaultsTest {

    @Test
    fun everyPlatformReportsTheSharedDefaults() = runTest {
        val config = RemoteConfig(testSdk(fetchFails = true))   // force the default path

        ConfigDefaults.values.forEach { (key, expected) ->
            val actual: Any = when (expected) {
                is ConfigValue.Bool  -> config.boolean(key).value
                is ConfigValue.Int64 -> config.long(key).value
            }
            val want: Any = when (expected) {
                is ConfigValue.Bool -> expected.v; is ConfigValue.Int64 -> expected.v
            }
            assertEquals(want, actual,
                "'$key' defaults to $actual on this target and $want in ConfigDefaults — " +
                "a platform declaring its own default is how 10 and 25 coexisted")
        }
    }

    @Test
    fun aDefaultSourcedReadIsReportedAsSuch() = runTest {
        val config = RemoteConfig(testSdk(fetchFails = true))
        assertEquals(ConfigSource.LocalDefault, config.long("free_note_limit").source)
    }

    @Test
    fun anUnknownKeyIsMissingRatherThanFalse() = runTest {
        // The typo case: previously `false` with no error, which silently disables a feature.
        assertEquals(ConfigSource.Missing, RemoteConfig(testSdk()).boolean("paywal_enabled").source)
    }
}

And the output it produced:

$ ./gradlew :core:config:allTests
everyPlatformReportsTheSharedDefaults     PASSED  × 5 targets
aDefaultSourcedReadIsReportedAsSuch       PASSED  × 5 targets
anUnknownKeyIsMissingRatherThanFalse      PASSED  × 5 targets

$ # against the four original maps
everyPlatformReportsTheSharedDefaults     FAILED  (android)
  'free_note_limit' defaults to 10 on this target and 25 in ConfigDefaults
everyPlatformReportsTheSharedDefaults     FAILED  (desktop)
  'free_note_limit' defaults to 50 on this target and 25 in ConfigDefaults
everyPlatformReportsTheSharedDefaults     FAILED  (web)
  'free_note_limit' defaults to 50 on this target and 25 in ConfigDefaults

$ ./tools/config-source-report.sh --days 9
  billing decisions made on a LocalDefault
    before   1,340
    after       19      (the 800 ms wait resolves the rest; 19 were genuinely offline)
  cross-platform disagreement on the same account
    before   1,340
    after        0

How we proved it#

That the four defaults disagreed — the measurement nobody had taken, because each map was reasonable where it was written:

  key                android    ios     desktop    web
  paywall_enabled      true     false    false     false
  free_note_limit        10        25       50       50
  sync_interval_min      15        15       30       15

Three of three keys disagreed on at least one platform.

That the default path is the normal first-launch path, which inverts the intuition that made this survive:

  cold start → the note-limit check          ~400 ms
  cold start → fetch complete, p50            310 ms
  cold start → fetch complete, p90            1.2 s
  checks that ran before the fetch, 9 days    1,340 of 3,900 first launches  (34%)

A third of first launches decided on a default. Defaults are not an edge case here.

That a bare Boolean return hid it. Established by what the caller could do, because this is the API-shape half of the fix:

  question                                    answerable from boolean(key)?
  "is the paywall on?"                                 yes
  "did that come from the server?"                     no
  "should I wait before billing on it?"                no

That the typo case was real and silent. Found by the Missing source rather than by a failure:

$ grep -rn 'config\.\(boolean\|long\)("' --include='*.kt' . | wc -l
41
$ ./gradlew :core:config:verifyConfigKeys
FAILURE: 2 config read(s) use a key absent from ConfigDefaults:
  FeedScreen.kt:88   "paywal_enabled"        ← a typo, reading as false for 7 months
  SyncWorker.kt:34   "sync_interval_minutes" ← renamed server-side; the old key still read

Two of forty-one call sites were reading keys that do not exist, both returning false/0 with no error. One had been disabling a feature for seven months.

That the bounded wait at the decision point is cheaper than a startup gate. Both measured, since the startup gate is the correct-looking fix:

  approach                       added startup latency   billing decisions on a default
  no wait                                0 ms                      1,340
  await fetch at startup              p99 4,100 ms                     0   (+ 2.9% web hang)
  800 ms wait at the paywall check       0 ms                         19

And the Double round-trip, which we found while writing the shared map. Not a bug we had, and a bug we would have had:

  default declared     iOS numberValue()       .toLong()
  25L                  25.0                    25          ok
  9_007_199_254_740_993L  9.007199254740992E15  …992        off by 1

Our largest config value is 50, so this was never wrong. The require is there because the next one might be a timestamp.

What I cannot tell you is whether "fail open" is right for every value. It is right for the paywall — wrongly charging is worse than wrongly not charging — and the same reasoning gives the opposite answer for a safety-related flag, which we do not have. The decision is per key and currently recorded as a comment rather than enforced.

And an honest gap. ConfigSource tells a caller where a value came from and not whether it is current. A fetched value can be hours stale — the SDKs cache with a minimum fetch interval — so Fetched means "came from a server response at some point", and nothing in our API reports its age.

What this generalizes to#

When a shared value has a per-platform fallback, the fallback is the part that will diverge, because it lives in the platform code that nothing compares. And a read that cannot report its source makes the divergence invisible: the caller gets a plausible value and no way to ask whether it is the configured one. The general shape is to declare the fallback once in shared code, have every platform consume it, and return the provenance alongside the value — because the decisions that matter are exactly the ones you would make differently if you knew you were guessing.

---

Related#

• A flag read is a sample, not a constant — the same config layer, where the defect is reading the value several times rather than defaulting it.
• A staged rollout you control on one store and not on the others — what the kill switch this config powers can and cannot halt.

Keep reading#

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.