ignoreUnknownKeys is a compatibility policy you are choosing for everyone

TL;DR#

• ignoreUnknownKeys = false took the app down for 40 minutes when the server added one optional field. Six weeks later a renamed field was silently dropped and 1,100 orders recorded a zero discount — which the setting had nothing to do with, in either position.
• Both settings are a compatibility policy. Strict means a server addition is a client outage; lenient means a server rename is silent data loss. There is no safe default and the flag looks like a convenience.
• What we ship now is lenient parsing plus an unknown-key report, and no defaults on the fields whose absence is a defect — so an addition is survivable and a disappearance is loud.
• What reading on buys: the two failure modes measured, the three-line reporting hook, and which fields earn @Required.

The case this post is about#

An order DTO and a strictly-configured parser:

// commonMain/kotlin/order/OrderDto.kt
@Serializable
data class OrderDto(
    val id: String,
    val total: Long,
    // Defaults, added early so a partially-populated test fixture was easy to write. Nobody
    // thought of them as a compatibility policy. They are the whole second half of this post.
    val discountMinor: Long = 0,
    val currency: String = "GBP",
    val lines: List<LineDto> = emptyList(),
)

// commonMain/kotlin/network/Json.kt
val json = Json {
    // the default, and a deliberate choice at the time: "fail loudly on an unexpected shape"
    ignoreUnknownKeys = false
    encodeDefaults = true
}

The reasoning was sound and I would make the same argument today: a response we do not understand is a response we should not act on.

The symptom#

The strict failure. The server added "loyaltyTier": "gold" to the order response, on a Tuesday afternoon, as an additive and backwards-compatible change:

kotlinx.serialization.json.internal.JsonDecodingException:
  Unexpected JSON token at offset 284: Encountered an unknown key 'loyaltyTier'.
  Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.

Every order screen on every platform, for 40 minutes, until the server rolled back. From the server team's point of view they had added an optional field to a JSON response — which is the canonical example of a safe change.

The lenient failure. We set ignoreUnknownKeys = true the same day. Six weeks later the server renamed discountMinor to discountAmountMinor as part of a currency refactor, with the old field kept for one release and then removed:

# no error, anywhere
  orders parsed                     1,100
  orders with discountMinor present     0
  discount shown to the user          £0.00
  orders where a discount applied   1,100

discountMinor was absent, so the property took its declared default of 0, and a zero discount renders as £0.00 like any other. Eleven hundred order confirmations showing no discount, and the users who noticed contacted support about the price rather than about the discount.

The flag flip did not cause this, which took me an embarrassingly long time to accept. A property with a default takes that default on a missing key whatever ignoreUnknownKeys is set to. Had we never touched the setting, the rename would have produced the same 1,100 zeroes — the strict configuration had never been protecting this direction at all.

Why the obvious fix failed#

The first fix was to make the field nullable so its absence is at least representable:

val discountMinor: Long? = null

Which moves the decision to every call site and most call sites pick the convenient default:

val discount = dto.discountMinor ?: 0L       // ← in 4 of 6 places

Same zero, now written by hand four times. Nullable types make absence expressible and do not make it handled.

The second fix was to drop the default so deserialization fails on a missing field:

val discountMinor: Long       // no default

kotlinx.serialization.MissingFieldException: Field 'discountMinor' is required

Which is correct for this field and, applied as a policy, is the strict failure again in a different dress — every field without a default becomes a field whose removal is an outage, and some fields genuinely are optional.

The third attempt was schema validation ahead of parsing, which is a real approach and more than we needed: it requires maintaining a schema alongside the DTOs, and the thing we actually wanted was to notice unknown and missing keys rather than to reject on them.

The actual mechanism#

ignoreUnknownKeys governs one direction of a two-directional compatibility problem, and the two directions have opposite safe defaults.

A server ADDS a field. The client receives a key it has no property for. Strict parsing rejects the whole response; lenient parsing ignores the key. Here lenient is correct — an additive change is the most common kind of server change and it cannot be coordinated with every deployed client.

A server REMOVES or RENAMES a field. The client's property has no incoming value. If it has a default, it silently takes it; if it does not, deserialization fails. Here strict is correct for any field the application depends on — and ignoreUnknownKeys has no bearing on this case at all, which is the part that confuses the discussion. It is a different mechanism: presence of a default on the property.

So the two settings people argue about are controlling different things:

                        server adds a field      server removes a field
  ignoreUnknownKeys       ← governs this          no effect
  a property default      no effect               ← governs this

Which means the right configuration is not a choice between them. Lenient on unknown keys (because additions are normal and uncoordinated) plus no default on fields the app depends on (because their absence is a defect). The two are independent and we had been treating one flag as though it covered both.

And the missing third piece is observability. Lenient parsing is silent by design, so an addition we ignore is an addition we never learn about — including the one that was the first half of the rename. The server added discountAmountMinor six weeks before removing discountMinor, and a report of unknown keys would have shown us the new field during the overlap window, which is exactly when a migration is free.

The fix#

Lenient on unknown keys, loud about them, and strict on the fields that matter.

// commonMain/kotlin/network/Json.kt
val json = Json {
    // Lenient, because an additive server change cannot be coordinated with deployed clients
    // and must not be an outage. This is the LESS dangerous half of the configuration.
    ignoreUnknownKeys = true

    // Unrelated to compatibility, and here because they are part of this parser's config.
    classDiscriminator = "type"
    explicitNulls = false
}

// There is no Json option that reports what lenient parsing ignored — the setting is silent by
// construction. That is why the reporting below is a WRAPPER rather than a flag, and it is the
// piece that makes the first half of a rename visible during the overlap window when migrating
// is still free.

/**
 * Wraps a decode and reports keys the DTO had no property for. Sampled rather than logged
 * per response: an unknown key appears on EVERY response once the server adds it, so logging
 * each one is a log-volume incident of its own.
 */
inline fun <reified T> Json.decodeAndReport(raw: String, endpoint: String): T {
    val value = decodeFromString<T>(raw)

    if (unknownKeyReporter.shouldSample(endpoint)) {
        val seen = parseToJsonElement(raw).keysDeep()
        val known = serializersModule.serializer<T>().descriptor.elementNamesDeep()
        val unknown = seen - known
        if (unknown.isNotEmpty()) {
            // A FACT, not a warning: the server added fields and this client does not read
            // them. Which is fine, and is the signal that a migration window is open.
            analytics.log("json.unknown_keys",
                "endpoint" to endpoint, "keys" to unknown.sorted().joinToString(","))
        }
    }
    return value
}

// commonMain/kotlin/order/OrderDto.kt
@Serializable
data class OrderDto(
    // No default is what makes a removal a MissingFieldException rather than a zero — that
    // is the mechanism, applied per field rather than per parser. @Required adds nothing on a
    // property that already has no default, and it is here on purpose anyway: it is the guard
    // against the next person adding `= 0` for a convenient test fixture, which is precisely
    // how the defaults above got there in the first place.
    @Required val id: String,
    @Required val total: Long,
    @Required val currency: String,

    // Genuinely required: a missing discount is not "no discount", it is an unknown discount,
    // and showing £0.00 for it cost 1,100 confirmations.
    @Required val discountMinor: Long,

    @Required val lines: List<LineDto>,

    // Genuinely optional, with a default that is CORRECT rather than convenient: this field
    // did not exist before the loyalty launch and its absence means the customer has no tier.
    val loyaltyTier: String? = null,
)

And the test that pins both directions, because the configuration is two independent decisions and a change to either is silent:

// commonTest/kotlin/order/OrderDtoCompatibilityTest.kt
class OrderDtoCompatibilityTest {

    /** A server ADDITION must not break the client. The 40-minute outage, as a test. */
    @Test
    fun anUnknownFieldIsIgnored() {
        val withNewField = """
            {"id":"o1","total":2400,"discountMinor":200,"currency":"GBP","lines":[],
             "loyaltyTier":"gold","somethingAddedTomorrow":{"nested":true}}
        """.trimIndent()

        val dto = json.decodeFromString<OrderDto>(withNewField)
        assertEquals("o1", dto.id)
    }

    /** A server REMOVAL must break the client. The 1,100 zero-discounts, as a test. */
    @Test
    fun aMissingRequiredFieldFails() {
        val withoutDiscount = """
            {"id":"o1","total":2400,"currency":"GBP","lines":[]}
        """.trimIndent()

        val e = assertFailsWith<MissingFieldException> {
            json.decodeFromString<OrderDto>(withoutDiscount)
        }
        assertTrue("discountMinor" in e.message.orEmpty(),
            "a removed discount must fail loudly; a default here records £0.00 for an " +
            "unknown discount, which is what 1,100 order confirmations did")
    }

    /** And the optional field must stay optional, or we have reintroduced the outage. */
    @Test
    fun aGenuinelyOptionalFieldMayBeAbsent() {
        val dto = json.decodeFromString<OrderDto>(
            """{"id":"o1","total":2400,"discountMinor":0,"currency":"GBP","lines":[]}"""
        )
        assertNull(dto.loyaltyTier)
    }
}

And the output it produced — replaying both historical payloads against the new configuration:

$ ./gradlew :core:order:allTests

anUnknownFieldIsIgnored            PASSED   × 5 targets
aMissingRequiredFieldFails         PASSED   × 5 targets
aGenuinelyOptionalFieldMayBeAbsent PASSED   × 5 targets

$ # and the unknown-key report, against production for a week
$ ./tools/unknown-key-report.sh --days 7
  endpoint        unknown keys
  GET /orders     loyaltyTier, discountAmountMinor        ← the rename's first half, visible
  GET /profile    marketingConsentV2
  GET /feed       (none)

discountAmountMinor appearing in that report is the signal the original configuration could not produce — and it appeared six weeks before the old field went away.

How we proved it#

That both failures are real and opposite — replayed from the actual payloads:

  payload                            strict (as it was)   lenient (as we changed it)
  with loyaltyTier added             JsonDecodingException  parses
  with discountMinor removed         parses, discount = 0   parses, discount = 0   ← identical
  the shipped configuration          parses the addition, throws on the removal

The middle row is the point, and it is the row I had not predicted: both settings are silent on the removal, because the default decides that case and neither setting touches the default. The shipped configuration is the only one that gets both directions right, and it does it with two independent mechanisms rather than one flag.

That ignoreUnknownKeys has no bearing on a removal. Worth demonstrating, because the two were conflated in every discussion we had:

  ignoreUnknownKeys   property has a default   field removed →
        false                 yes              parses, takes the default
        true                  yes              parses, takes the default
        false                 no               MissingFieldException
        true                  no               MissingFieldException

The flag is irrelevant to the removal case in all four rows. The default is what decides it.

That nullable did not help. Counted across the call sites, because "make it nullable" was the first fix and reads as more careful:

  dto.discountMinor ?: 0L          4 of 6 call sites
  explicit absent-handling         2 of 6

Four of six reproduced the zero by hand. Expressible is not handled.

That the unknown-key report would have caught the rename. Checked against the server's own changelog rather than assumed:

  server release 2.31   adds discountAmountMinor  (both fields present)      ← 6 weeks of overlap
  server release 2.34   removes discountMinor
  our unknown-key report, replayed over 2.31 payloads:
    GET /orders  unknown keys: discountAmountMinor

Six weeks of warning, available from a three-line hook, discarded by a setting whose entire purpose is to discard it quietly.

That sampling the report is necessary. The naive version logged per response and we noticed immediately:

  log lines / day, unsampled      1.4M      (one per order response, per unknown key)
  log lines / day, sampled 1:500  2.8k
  unknown keys discovered         identical

An unknown key is present on every response once the server adds it, so the information is entirely in the first occurrence.

Which fields got @Required, and the judgement behind it. This is the part a gate cannot decide:

  field              @Required   reason
  id                   yes       nothing works without it
  total                yes       a zero total is a wrong price, not a free order
  currency             yes       a default currency is a wrong currency
  discountMinor        yes       absence means unknown, not none        ← the 1,100
  lines                yes       an empty order is a different thing from an unparsed one
  loyaltyTier          no        predates the feature; absence genuinely means no tier

Five of six are required, which surprised me — I had expected most fields to be legitimately optional, and on inspection almost none were. The test is whether the default would be correct or merely convenient.

What I cannot tell you is whether keysDeep() covers every nesting case. It walks objects and arrays and I have not exercised it against polymorphic payloads with a discriminator, where the known-key set depends on the resolved subtype. The report is a best-effort signal rather than a guarantee, and treating it as the latter would be the same mistake one level up.

And an honest residual. @Required turns a field removal into a parse failure, which is an outage — a deliberate one, on the theory that showing a wrong price is worse than showing an error. For discountMinor I am confident. For lines I am less sure: an order with unparseable lines might be better shown partially than not at all, and we chose the strict option without a strong argument.

What this generalizes to#

A flag that reads as a convenience is often a policy, and the test is whether flipping it changes which failures you get rather than how much code you write. Here one setting was being asked to govern two independent compatibility directions with opposite safe answers, so neither value was right — and the resolution was to stop treating it as a choice: lenient where additions arrive, strict per field where absence is a defect, and a report in between so the silent half is not silent. Before setting any tolerance, ask what it will let through and whether you will hear about it.

---

Related#

• A @Serializable class's declaration is its wire contract — the other way an innocuous-looking edit changes what parses.
• The null that meant three things — why making absence expressible is not making it handled.

Keep reading#

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.