R8 stripped a reflective call that only the shared module made

TL;DR#

• A @Serializable polymorphic hierarchy that worked on four targets and threw SerializationException: Class 'PushPayload' is not registered on release Android only — because R8 had removed a subclass nothing referenced statically.
• Shrinking runs on the Android target and nowhere else. So a reflection-dependent path in commonMain is exercised on four targets that do not shrink and ships broken on the one that does.
• Four targets green plus one release-only failure is the shape, and the four green ones are actively misleading: they prove the code is correct and say nothing about whether it survives.
• What reading on buys: why commonTest cannot catch it, the keep rules that are actually needed, and the gate that fails a release whose shrunk output has lost a registered subtype.

The case this post is about#

A push payload, modelled polymorphically in shared code:

// commonMain/kotlin/push/PushPayload.kt
@Serializable
sealed interface PushPayload {
    @Serializable @SerialName("message")  data class NewMessage(val threadId: String) : PushPayload
    @Serializable @SerialName("mention")  data class Mention(val messageId: String) : PushPayload
    @Serializable @SerialName("digest")   data class DailyDigest(val count: Int) : PushPayload
}

// commonMain — the dispatch, which is where the subclasses are reached
suspend fun handlePush(raw: String) {
    when (val p = json.decodeFromString<PushPayload>(raw)) {
        is PushPayload.NewMessage  -> openThread(p.threadId)
        is PushPayload.Mention     -> openMessage(p.messageId)
        is PushPayload.DailyDigest -> showDigest(p.count)
    }
}

Every subtype is named in that when, so every one is statically reachable. Which is true of the types and, as it turns out, not of what deserialization needs.

The symptom#

A crash, Android release builds only:

kotlinx.serialization.SerializationException: Class 'digest' is not registered for
  polymorphic serialization in the scope of 'PushPayload'.
  Mark the base class as 'sealed' or register the serializer explicitly.
    at com.example.push.PushPayload$Companion.serializer(Unknown Source:0)

The base class is sealed. And the state of each build:

  debug android       works
  release android     throws
  ios release         works
  desktop release     works
  wasm release        works
  commonTest          passes, on all five targets

One configuration out of six, and the five that worked included the entire shared test suite — which runs unshrunk, because commonTest executes against unminified output on every target.

The specific subtype was DailyDigest, and the reason it and not the others is the thing that made this hard: the digest push was the only payload whose handler was behind a feature flag that was off at the time of the release build, so its when branch was the only one R8 could prove unreachable.

Why the obvious fix failed#

The first fix was a keep rule on the hierarchy, which is the documented shape:

-keep class com.example.push.PushPayload { *; }
-keep class com.example.push.PushPayload$* { *; }

Still threw. -keep preserves the classes and their members, and kotlinx.serialization's polymorphic resolution needs the synthetic $serializer nested class and its Companion, which are generated by the compiler plugin and are not matched by PushPayload$* in the way the author expects — $* matches nested classes of PushPayload, and NewMessage$$serializer is nested inside NewMessage, one level deeper.

The second fix was to disable shrinking:

buildTypes { release { isMinifyEnabled = false } }

Which works and grew the APK by 4.2 MB, and gives up the one thing minification was doing for us. It also leaves the next reflection-dependent path to fail the same way whenever somebody re-enables it.

The third fix was -dontwarn plus a broader keep, which I include because it is where I spent an hour on the wrong theory:

-keep,includedescriptorclasses class com.example.** { *; }

Keeping the whole application package. The crash stopped, the APK grew by 3.8 MB, and the rule preserves tens of thousands of members to fix a problem with three. A keep rule that keeps everything is isMinifyEnabled = false with extra steps.

The actual mechanism#

Shrinking is a per-target build step, and only one target has it.

R8 runs on the Android target only. It takes the whole dex — your Android code, your shared Kotlin, the stdlib, every library — and removes what it cannot prove is reachable. The input includes commonMain's output, so shared code is shrunk by a step that no other target applies.

Reachability is a static-analysis question and reflection is invisible to it. A class resolved by name at runtime has no static reference. kotlinx.serialization is a compiler plugin, so most of it is static — the generated $serializer is referenced from the generated Companion, which is referenced from the type. But polymorphic deserialization resolves a subtype from a discriminator string at runtime, and that lookup goes through a registry built from the sealed hierarchy's serializers. If a subtype's own code is unreachable, R8 removes the subtype, and the registry entry it would have contributed goes with it.

So the reachability of the when branch decided it. NewMessage and Mention had live handlers; DailyDigest's branch called a function behind a disabled flag, and R8 traced that to nothing, so it removed DailyDigest and its serializer. The shared code's source still had three subtypes and the shipped dex had two.

And commonTest structurally cannot catch it. Tests run against unshrunk output — that is true on every target including Android, where unit tests run against the unminified classes. So a reflection-dependent path has a test suite that proves it correct and no test that proves it survives, and the four non-shrinking targets add four more green signals that mean nothing about the failing configuration.

The fix#

Narrow keep rules naming the serializers, generated from the hierarchy rather than hand-listed, and a gate that inspects the shrunk output.

# shared/proguard-rules.pro
#
# NARROW, and aimed at what polymorphic resolution actually needs: the generated $serializer
# of every subtype, plus the Companion that exposes it. `-keep class …PushPayload$* { *; }`
# does NOT cover these — $* matches classes nested in PushPayload, and the serializer is
# nested one level deeper, inside each subtype.

# Every @Serializable type's generated serializer, by shape rather than by name — so a new
# subtype is covered without anyone remembering to add a line.
-if @kotlinx.serialization.Serializable class **
-keepclassmembers class <1> {
    static <1>$Companion Companion;
}
-if @kotlinx.serialization.Serializable class ** {
    static **$* *;
}
-keepclassmembers class <2>$<3> {
    kotlinx.serialization.KSerializer serializer(...);
}
-keep,includedescriptorclasses class **$$serializer { *; }

# And the sealed subtypes themselves, which R8 removes when their only reference is an
# unreachable `when` branch. Scoped to the serialization package rather than the app.
-keep,allowobfuscation,allowoptimization @kotlinx.serialization.SerialName class com.example.**

And the gate, because a keep rule is only correct until the next subtype:

// build-logic/src/main/kotlin/VerifyShrunkSerializers.kt
//
// Reads the SHRUNK dex and asserts that every @SerialName-annotated subtype in the source is
// still present. A keep rule is a claim about the output, and the only way to check a claim
// about the output is to read the output — which is also why commonTest cannot do this.
abstract class VerifyShrunkSerializers : DefaultTask() {

    @get:InputFile      abstract val mappingFile: RegularFileProperty      // R8's mapping.txt
    @get:InputFiles     abstract val sharedSources: ConfigurableFileCollection
    @get:InputFile      abstract val dexFile: RegularFileProperty

    @TaskAction
    fun verify() {
        // Every subtype the SOURCE declares, by its discriminator — which is what the runtime
        // registry is keyed on and therefore what must survive.
        val declared = sharedSources.files
            .filter { it.extension == "kt" }
            .flatMap { f ->
                Regex("""@SerialName\("([^"]+)"\)\s*(?:data\s+)?(?:class|object)\s+(\w+)""")
                    .findAll(f.readText())
                    .map { it.groupValues[1] to it.groupValues[2] }
            }.toMap()

        // The mapping file lists every type R8 KEPT, under its original name. A type absent
        // from the mapping was removed — which is the fact no test can observe.
        val kept = mappingFile.get().asFile.readLines()
            .mapNotNull { Regex("""^([\w.$]+) ->""").find(it)?.groupValues?.get(1) }
            .map { it.substringAfterLast('.') }
            .toSet()

        val missing = declared.filterValues { it !in kept }
        if (missing.isNotEmpty()) {
            error(buildString {
                appendLine("R8 removed ${missing.size} @Serializable subtype(s) from the release build:")
                missing.forEach { (discriminator, cls) ->
                    appendLine("  $cls (discriminator \"$discriminator\")")
                }
                appendLine()
                appendLine("Polymorphic deserialization resolves these by discriminator at RUNTIME,")
                appendLine("so R8 cannot see the reference and removes the subtype when its only")
                appendLine("static use is an unreachable branch. The failure is a release-only")
                appendLine("SerializationException that no test on any target will catch, because")
                appendLine("tests run against unshrunk output everywhere.")
            })
        }
        logger.lifecycle("${declared.size} serializable subtype(s) survived shrinking")
    }
}

And the output it produced, run against the release build that was crashing:

$ ./gradlew :androidApp:verifyShrunkSerializers

FAILURE: R8 removed 1 @Serializable subtype(s) from the release build:
  DailyDigest (discriminator "digest")

Polymorphic deserialization resolves these by discriminator at RUNTIME, so R8 cannot
see the reference and removes the subtype when its only static use is an unreachable
branch. The failure is a release-only SerializationException that no test on any
target will catch, because tests run against unshrunk output everywhere.

$ # after the keep rules
$ ./gradlew :androidApp:verifyShrunkSerializers
41 serializable subtype(s) survived shrinking
BUILD SUCCESSFUL

And the size, which is the thing the narrow rules were for:

  isMinifyEnabled = false              APK 9.4 MB
  keep com.example.** { *; }           APK 9.0 MB
  the narrow serializer rules          APK 5.3 MB

How we proved it#

That it was shrinking and not Android — the measurement that separates the two, because "Android release only" has two candidate causes:

  configuration                              result
  android debug                              works
  android release, isMinifyEnabled = true    throws
  android release, isMinifyEnabled = false   works
  android release, minify on, keep rules     works

The second and third rows differ in one flag with identical source, which excludes everything about the Android platform itself.

That commonTest cannot catch it, on any target. Worth establishing rather than asserting, because "add a test" is the instinct:

  test run                                 runs against          caught it?
  commonTest → jvmTest                     unshrunk classes          no
  commonTest → androidUnitTest             unshrunk classes          no
  commonTest → iosSimulatorArm64Test       no shrinking exists       no
  androidTest (instrumented, debug)        unshrunk dex              no
  androidTest (instrumented, release)      SHRUNK dex                YES

Only the last row, which most projects do not run — and it is the only test configuration whose input is the artifact that ships.

That the feature flag decided which subtype was removed. The control, because it explains why one of three:

  flag state at build time     subtypes R8 kept
  digest flag ON               NewMessage, Mention, DailyDigest
  digest flag OFF              NewMessage, Mention

Flipping a flag changed which classes exist in the artifact. That is R8 working correctly and it means the set of surviving serializers is a function of build-time configuration.

That -keep class PushPayload$* { *; } does not cover the serializer. Read from the mapping rather than inferred, because this rule looks right and is the one everyone writes first:

  rule                                        DailyDigest kept   DailyDigest$$serializer kept
  -keep class …PushPayload$* { *; }                 yes                    no
  -keep class **$$serializer { *; }                 no                     yes
  both                                              yes                    yes

Two separate rules for two separate things, and either alone still throws.

That the broad rule costs 3.7 MB. Measured, since "just keep the package" is the pragmatic temptation:

  the narrow serializer rules      APK 5.3 MB   members kept:     ~2,400
  keep com.example.** { *; }       APK 9.0 MB   members kept:  ~218,000

Ninety times the members, to fix a problem with one class.

What I cannot tell you is whether the -if-based rules cover every serialization shape. They cover sealed hierarchies, enums and plain @Serializable classes in our corpus — 41 subtypes across 9 hierarchies — and a custom serializer registered through a SerializersModule has no annotation for the rule to match, so it would need its own keep line. We have one of those and it is listed explicitly with a comment.

And an honest limitation of the gate. It reads mapping.txt, which R8 only produces when shrinking is enabled — so the gate silently passes on a build with minification off. It asserts mappingFile exists as an input, so a release build without it fails at configuration rather than passing vacuously, and that is the only thing standing between this check and the vacuous-green class it exists to prevent.

What this generalizes to#

A build step that runs on one target applies to all the code that target consumes, including the shared code the other targets compile differently — so "shared" does not mean "identically processed". The dangerous version is a step that removes things, because its effect is invisible to every test that runs against the unprocessed artifact, and tests almost always do. Where behaviour depends on something static analysis cannot see, the only honest check reads the shipped artifact rather than the source, and the shipped artifact differs per target.

---

Related#

• A klib is not a jar — the other case where the artifact that shipped and the source that compiled disagree.
• A @Serializable class's declaration is its wire contract — what the plugin generates, and why most of it is static.

Keep reading#

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.