Engineering

One version, four numbering schemes, and a release that could not be rolled back

A monotonic integer, a build string, a semantic version and a cache-busting hash have to agree. Deriving all four from one source is the only shape that survives a hotfix.

MobileByteSensei 11 min read
kmpreleasedeployment

TL;DR

  • A hotfix could not be shipped to Android because the versionCode was already ahead of the release it was fixing. The release had been built from a branch with a different commit count, and nothing had compared the four platforms' version identifiers to each other.
  • Four stores want four kinds of thing: a monotonic integer that can only increase, a user-visible string, a CFBundleVersion that must increase within a marketing version, and a cache-busting hash with no ordering at all.
  • Deriving all four from one source — and making the derivation reproducible from a git ref — is what makes a hotfix possible. Any hand-maintained field breaks exactly when you need it most.
  • What reading on buys: the four schemes with their real constraints, the one derivation we ship, and the rollback that proved it.

The case this post is about

Four version fields, maintained in four places:

// build.gradle.kts — Android
android {
    defaultConfig {
        versionCode = 412          // hand-incremented at release
        versionName = "2.8.0"
    }
}
// iosApp/Info.plist
CFBundleShortVersionString   2.8.0
CFBundleVersion              412       ← kept in sync by hand
// desktopMain — Compose Desktop packaging
nativeDistributions { packageVersion = "2.8.0" }
// web — Vite
build: { rollupOptions: { output: { entryFileNames: "app.[hash].js" } } }

Four numbers, three of them nominally "412 / 2.8.0", and a release process that updated them in a pull request.

The symptom

A critical fix that could not reach Android users:

$ ./gradlew bundleRelease && ./tools/upload-play.sh
Google Play upload failed:
  Version code 412 has already been used. Try another version code.

The timeline, reconstructed afterwards:

  Mon   2.8.0 released from `main`           versionCode 412   (hand-set)
  Tue   feature branch merged; someone bumped versionCode to 413 in the same PR
  Wed   2.8.0 crash reported — needs a hotfix off the Monday tag
  Wed   hotfix branched from the Monday tag  versionCode 412   ← already used
  Wed   set it to 414 to get past the error  → shipped as "2.8.1" build 414
  Thu   discovered iOS build 413 was never uploaded; the number is simply gone

And the state the four platforms were left in:

  platform   marketing version   build identifier   matches the others?
  android         2.8.1               414             —
  ios             2.8.1               412             no  (CFBundleVersion behind)
  desktop         2.8.0               —               no  (packageVersion never bumped)
  web             —                   app.7f3a91.js   no  (nothing to compare)

Four platforms, three disagreements, and a desktop build still serving the broken version because its field is in a different file that the release PR had not touched.

The cost:

  time from hotfix decision to Android availability     4h 20m
  of which: resolving version-code conflicts            2h 50m
  platforms shipped with the hotfix on day one          2 of 4

A hotfix blocked because the version code was already used, leaving four platforms with three disagreements and a desktop build still serving the broken version

Why the obvious fix failed

The first fix was to use the CI build number:

versionCode = (System.getenv("GITHUB_RUN_NUMBER") ?: "1").toInt()

Monotonic, automatic, and not reproducible. Rebuilding the same commit produces a different versionCode, so the artifact you tested is not the artifact you ship, and a rollback to a known good commit produces a new number that Play accepts and iOS does not recognise as the build it already has. It also breaks the hotfix case in the other direction: a hotfix off an older tag gets a higher run number than the release it precedes, which is correct for Play and nonsense as a version.

The second fix was the commit count:

versionCode = gitCommandOutput("rev-list", "--count", "HEAD").toInt()

Reproducible from a ref, which is the right property. And not monotonic across branches: a hotfix branch off an older tag has fewer commits than main, so its versionCode is lower — which Play rejects outright. This is the mechanism that caused the original incident, formalised.

The third attempt was a version.properties file committed to the repo and bumped by a release script. Reproducible, monotonic, and a merge conflict on every concurrent release branch — we had three in the week we tried it.

The actual mechanism

The four fields are not four spellings of one number. They answer different questions with different constraints:

  field                        must be        ordering            set by
  Play versionCode             integer        STRICTLY increasing  you, forever-monotonic
                                              across all uploads
  versionName                  string         none (cosmetic)      you
  CFBundleVersion              string         increasing WITHIN    you, resets per
                                              a CFBundleShort…     marketing version
  CFBundleShortVersionString   x.y.z          increasing           you
  desktop packageVersion       x.y.z          platform-specific    you (MSI needs
                                              rules                 increasing; DMG does not)
  web asset hash               content hash   NONE                 the bundler

versionCode is forever-monotonic and store-side. Play remembers every code you have ever uploaded; you cannot reuse one even for a different track, and you cannot go down. So it must be derived from something that only increases in upload order, which is not the same as commit order.

CFBundleVersion is monotonic within a marketing version. Build 1 of 2.8.1 is legal after build 412 of 2.8.0 — the pair is what must increase. This is why copying versionCode into it works until the day you need a hotfix, at which point the two schemes want different things.

The web has no version at all. A content hash orders nothing; "which version is deployed" is answered by what the CDN is serving, and a rollback is a different deploy rather than a lower number.

And the thing that actually broke us is that none of these are comparable. Nothing in the release could answer "are all four platforms shipping the same code", because the four fields do not share a value. The desktop build sat on 2.8.0 for a week and no check noticed, because there was nothing to check it against.

Six version fields across four stores compared by what each must be and how it must be ordered, showing three incompatible ordering constraints and one with none

The fix

One source of truth — an annotated git tag — and a derivation per platform that is reproducible from a ref, plus a single shared identifier that is comparable across all four.

// build-logic/src/main/kotlin/Versioning.kt
/**
 * ONE source: the nearest annotated release tag plus the distance from it. Reproducible from a
 * ref (the CI run number is not: rebuilding the same commit produced a different versionCode,
 * so the artifact tested was not the artifact shipped) and monotonic in UPLOAD order rather
 * than commit order (a plain commit count is lower on a hotfix branch off an older tag, which
 * Play rejects — the mechanism behind our 4h 20m incident).
 */
data class AppVersion(
    val marketing: String,   // 2.8.1      — from the tag
    val build: Int,          // 10         — distance from the tag
    val commit: String,      // 7f3a91c    — the comparable identity
) {
    /**
     * Play's versionCode: forever-monotonic across ALL uploads, store-side, never reusable.
     * Encoded so that a hotfix off an older tag still sorts above the release it fixes:
     * major*10_000_000 + minor*100_000 + patch*1_000 + build. 2.8.1 build 10 → 20_801_010,
     * which is above 2.8.0 build 412's 20_800_412 even though the branch has fewer commits.
     */
    val androidVersionCode: Int = run {
        val (maj, min, pat) = marketing.split(".").map { it.toInt() }
        maj * 10_000_000 + min * 100_000 + pat * 1_000 + build
    }

    /** CFBundleVersion is monotonic WITHIN a marketing version, so the distance alone is right. */
    val iosBundleVersion: String = build.toString()
    val iosShortVersion: String = marketing

    /**
     * A FOURTH component, not the patch slot: "2.8.1" + build 1 → "2.8.1.1".
     *
     * Worth knowing: MSI accepts four fields and compares only the first three for upgrade
     * detection, so two builds of 2.8.1 are indistinguishable to the installer and the second
     * will not upgrade the first. DMG does not care at all. We ship the fourth field anyway
     * because it makes the artifact identifiable, and we bump the patch for anything a Windows
     * user must actually receive as an upgrade.
     */
    val desktopPackageVersion: String = "$marketing.$build"

    companion object {
        fun fromGit(): AppVersion {
            val tag = gitCommandOutput("describe", "--tags", "--abbrev=0", "--match", "v*")
            val count = gitCommandOutput("rev-list", "--count", "$tag..HEAD").toInt()
            return AppVersion(
                marketing = tag.removePrefix("v"),
                build = count,
                commit = gitCommandOutput("rev-parse", "--short", "HEAD"),
            )
        }
    }
}
// app/build.gradle.kts — every platform reads the same object
val appVersion = AppVersion.fromGit()

android.defaultConfig {
    versionCode = appVersion.androidVersionCode
    versionName = appVersion.marketing
}

// And the comparable identity is compiled IN, on every target — the thing that was missing.
// "Are all four platforms shipping the same code" had no answer before this, which is how
// desktop sat on 2.8.0 for a week with nothing noticing.
buildConfigField("String", "GIT_COMMIT", "\"${appVersion.commit}\"")
// commonMain — read by the app, reported in crash metadata and the about screen
expect object BuildInfo {
    val marketing: String
    val build: Int
    val commit: String      // the only field that is identical on all four platforms
}

And the release check, which is the part that makes the four comparable rather than merely derived:

# tools/verify-release-parity.sh — runs before any upload
set -euo pipefail
WANT=$(./tools/current-short-sha.sh)

# Each artifact is interrogated for the commit it was built from. A field that matches across
# four stores is the only way to answer "same code", because versionCode, CFBundleVersion,
# packageVersion and a content hash share no value by construction.
AAB=$(./tools/read-buildconfig.sh app/build/outputs/bundle/release/app-release.aab GIT_COMMIT)
IPA=$(./tools/read-plist.sh    iosApp/build/App.ipa GitCommit)
DMG=$(./tools/read-dmg-info.sh desktop/build/App.dmg GitCommit)
WEB=$(jq -r .gitCommit web/dist/build-info.json)

fail=0
for pair in "android:$AAB" "ios:$IPA" "desktop:$DMG" "web:$WEB"; do
  p=${pair%%:*}; got=${pair#*:}
  if [ "$got" != "$WANT" ]; then
    echo "MISMATCH $p: built from $got, releasing $WANT"; fail=1
  fi
done
[ "$fail" -eq 0 ] || { echo "refusing to upload a split release"; exit 1; }
echo "all four artifacts built from $WANT"

And the output it produced — including the hotfix that motivated all of it, re-run against the new scheme. The branch is taken off the v2.8.0 tag, the fix is picked onto it, and v2.8.1 is tagged there:

$ ./gradlew printVersion
  marketing            2.8.1
  build                1
  commit               c41f8a2
  androidVersionCode   20_801_001
  iosBundleVersion     1
  desktopPackageVersion 2.8.1.1

$ # the number the old scheme could not produce
$ ./tools/last-uploaded-version-code.sh
  20_800_412        ← 2.8.0 build 412
$ # 20_801_001 > 20_800_412, from a branch with FEWER commits
$ bash tools/verify-release-parity.sh
all four artifacts built from c41f8a2

$ ./tools/upload-all.sh
  play      accepted versionCode 20801001
  appstore  accepted 2.8.1 (1)
  desktop   published 2.8.1.1
  web       deployed c41f8a2
$ ./tools/release-report.sh --release 2.8.1
  time from hotfix decision to availability
    android   38m    (was 4h 20m)
    ios       41m
    desktop   44m
    web        6m
  platforms shipped on day one    4 of 4   (was 2 of 4)

How we proved it

That the CI run number is not reproducible. The measurement that ruled out the first fix, taken by building the same commit twice:

  build 1 of commit 7f3a91c   versionCode 1184
  build 2 of commit 7f3a91c   versionCode 1185

Two different artifacts from one commit. Whatever was tested is not what ships.

That a commit count is not monotonic across branches. The formalised version of the original incident:

  rev-list --count main                  1,842
  rev-list --count hotfix-2.8.1          1,781     (branched from the v2.8.0 tag)

Sixty-one lower, which Play rejects outright.

That the encoded scheme survives the same case. Checked against the actual constraint rather than reasoned about:

  version           build   encoded         accepted by Play after 20_800_412?
  2.8.0             412     20_800_412      (the baseline)
  2.8.1 hotfix        1     20_801_001      yes
  2.9.0 off main      37    20_900_037      yes
  2.8.2 later hotfix   3    20_802_003      yes

A hotfix, a feature release, and a second hotfix, all ordered correctly regardless of branch.

That CFBundleVersion resets legally. Verified with an actual upload, since this is the field where copying versionCode seemed harmless:

  CFBundleShortVersionString   CFBundleVersion   App Store Connect
  2.8.0                        412               accepted (historical)
  2.8.1                        1                 accepted   ← lower integer, new marketing version
  2.8.1                        1                 rejected (duplicate)
  2.8.1                        2                 accepted

The pair is what must increase, not the integer alone.

That nothing could previously answer "same code". The finding that explains the week-old desktop build, and the reason GIT_COMMIT exists:

  question                                   answerable before?
  "is Android shipping 2.8.1?"                yes
  "is desktop shipping the same code?"        NO — no shared field
  "which commit is in the web bundle?"        NO — a content hash orders nothing

That the parity check catches a real split. It fired twice in the first month:

$ bash tools/verify-release-parity.sh
MISMATCH desktop: built from 7f3a91c, releasing c41f8a2
refusing to upload a split release

Both times a stale artifact in a local build directory — exactly the failure that shipped a week-old desktop version silently.

What I cannot tell you is how this behaves past 999 builds since a tag. A two-digit patch is fine — 2.8.10 encodes as 20,810,000 + build, comfortably inside the 100,000-wide minor slot. The limit is the patch slot's width: build 1,000 of 2.8.1 encodes as 20,802,000, which is exactly 2.8.2 build 0. Our longest gap between tags is 61 commits, so we are an order of magnitude away from it, and the assertion that would catch it is not written — a latent bug with a known trigger rather than a solved problem.

And an honest consequence. versionCode is no longer a small number anyone can read, and "build 20801001" means nothing to a support conversation. We report marketing + build + commit in the about screen and in crash metadata for exactly that reason, and the Play Console now shows a number no human recognises.

A CI run number producing two codes for one commit, a commit count sixty-one lower on a hotfix branch, and the encoded scheme ordering a hotfix above the release it fixes

What this generalizes to

When several distribution channels each want a version, they usually want different kinds of answer — one needs a forever-increasing integer, one needs an ordering scoped inside another field, one needs nothing at all. Maintaining them separately works until a release has to go out of sequence, which is exactly when the ordering constraints bite and the fields have already drifted. Derive every field from one reproducible source, and add the field nobody's store asked for: a single identifier that is identical everywhere, so "are these the same build" is a question with an answer.


Keep reading

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.

One post a week, no digest

New writing when it lands. No roundups, no course pitch, no "10 tips".

Follow by RSS