A Kotlin crash on iOS arrives without a usable stack trace

TL;DR#

• A KotlinNullPointerException from a shared parser, in Crashlytics, as nine hexadecimal addresses. The Swift frames above and below it were symbolicated; only the Kotlin ones were not.
• The shared framework is a separate binary with its own build UUID, and its .dSYM is produced by Gradle into shared/build/bin/ — outside the Xcode archive that the upload script reads.
• The dSYM's UUID changes on every link, even from identical source, so a one-off manual upload fixes exactly one build and then silently stops working.
• What reading on buys: the UUID measurement, the upload wired into the build, and the gate that fails a release whose symbols were not uploaded.

The case this post is about#

A crash in shared code, reported from a release build:

Fatal Exception: KotlinNullPointerException
  0  Shared      0x00000001045a2f14  0x1045a0000 + 12052
  1  Shared      0x00000001045a1b88  0x1045a0000 + 7048
  2  Shared      0x00000001045b0210  0x1045a0000 + 66064
  3  Shared      0x00000001045b7c44  0x1045a0000 + 97860
  …
  9  iosApp      FeedViewModel.loadFeed()            FeedViewModel.swift:42
 10  iosApp      FeedView.body.getter                 FeedView.swift:18
 11  libswiftCore.dylib  …

Frames 9 and 10 are readable. Frames 0 through 8 — the entire Kotlin half, which is where the crash is — are offsets into a binary called Shared.

And the upload step in the release pipeline, which looked complete:

# .github/workflows/release-ios.yml
- name: Upload dSYMs
  run: |
    find "$ARCHIVE/dSYMs" -name '*.dSYM' -print0 \
      | xargs -0 "$FIREBASE_PATH/upload-symbols" -gsp GoogleService-Info.plist -p ios

The symptom#

The upload reported success and uploaded one file:

$ ls "$ARCHIVE/dSYMs"
iosApp.app.dSYM

$ # what the crash needs
$ dwarfdump --uuid "$ARCHIVE/Products/Applications/iosApp.app/Frameworks/Shared.framework/Shared"
UUID: 4F2A1C9E-7B3D-3E21-9A04-1C8E5B2D7F60 (arm64)

$ # and what Crashlytics holds
  dSYMs for this version:  1
    iosApp.app          B4D0E71A-…    ✓
    Shared.framework    (absent)

One dSYM in the archive, for the app. The framework's was not there — and find over the archive's dSYMs directory found exactly what was in it, so the step was behaving correctly and uploading everything it could see.

Crashlytics' own dashboard said so, in a place nobody looks:

  Missing dSYMs (1)
    4F2A1C9E7B3D3E219A041C8E5B2D7F60    Shared    arm64

A UUID with no name attached, in a list that had been growing for a year.

Why the obvious fix failed#

The first fix was to upload the one from the build directory, by hand:

$ "$FIREBASE_PATH/upload-symbols" -gsp GoogleService-Info.plist -p ios \
    shared/build/bin/iosArm64/releaseFramework/Shared.framework.dSYM
Successfully uploaded

That release's crashes symbolicated. The next release's did not, and the reason is the measurement that made this a build problem rather than a one-off:

$ dwarfdump --uuid shared/build/.../Shared.framework.dSYM
UUID: 4F2A1C9E-… (arm64)
$ ./gradlew clean && ./gradlew linkReleaseFrameworkIosArm64      # identical source
$ dwarfdump --uuid shared/build/.../Shared.framework.dSYM
UUID: B9041EE7-… (arm64)

A different UUID from an unchanged source tree. Crashlytics matches by UUID, so a stored dSYM matches exactly one build, and the manual step would have to be repeated — correctly — on every release forever.

The second fix was kotlin.native.binary.sourceInfoType=libbacktrace, which is a real setting for a different problem:

kotlin.native.binary.sourceInfoType=libbacktrace

That attaches source information to runtime stack traces, so e.stackTraceToString() inside the app becomes readable. It does nothing for a crash report, because the report is assembled by the SDK's signal handler from the crashed process and what it captures is addresses.

The third attempt was to disable the Kotlin framework's own symbol stripping, on the theory that the symbols were being removed:

binaries.framework { freeCompilerArgs += "-Xg-generate-debug-trampoline" }

The symbols were never in the binary — they are in the dSYM, which is the point of a dSYM. The binary was correctly stripped and the separate symbol file was correctly produced and nobody collected it.

The actual mechanism#

Symbolication is a UUID-keyed lookup, and the shared framework is a binary the Xcode archive pipeline does not produce.

The framework is linked by Gradle, not by Xcode. linkReleaseFrameworkIosArm64 invokes the Kotlin/Native linker, which emits Shared.framework and Shared.framework.dSYM into shared/build/bin/. Xcode then embeds the already-built framework. So Xcode's archive step — which collects dSYMs for everything it compiled — has nothing to collect for it.

Each Mach-O binary has its own build UUID, and a crash report records which UUID each frame's image had. The symbolicator needs the dSYM whose UUID matches. The app's dSYM contains no information about the framework's addresses, so the Kotlin frames are unresolvable with it.

The UUID is derived from the link, not from the source. Build inputs include paths, timestamps and the linker's own invocation, so two links of identical source produce different UUIDs. That is why a stored dSYM is per-build rather than per-version.

And an unresolved frame is not an error. A crash report with nine unsymbolicated frames is a valid crash report. Crashlytics lists the missing UUID in a dashboard section, the upload step exits zero, and nothing on either side fails — which is how it ran for a year.

So the gap is a build-wiring one: an artifact produced on every release, by a step outside the pipeline's field of view, needed by a system that fails silently without it.

The fix#

Collect the framework's dSYM into the archive as part of the build, upload it from the same step that uploads the app's, and gate the release on both having been uploaded.

// shared/build.gradle.kts
//
// Copies the dSYM Gradle produced into the location the Xcode archive step collects from, as a
// task the framework link finalises — so it happens on every release build rather than when
// somebody remembers. The alternative, an Xcode run-script phase, puts the knowledge in a
// place the Gradle build cannot see.
val collectFrameworkDsym by tasks.registering(Copy::class) {
    val linkTask = tasks.named<KotlinNativeLink>("linkReleaseFrameworkIosArm64")
    dependsOn(linkTask)

    from(linkTask.map { it.outputFile.get().parentFile.resolve("Shared.framework.dSYM") })

    // DWARF_DSYM_FOLDER_PATH is set by Xcode when it drives Gradle; when Gradle is driven
    // directly the destination is a build-local path and the archive step will not find it,
    // which is correct — a non-Xcode build has no archive.
    val dest = providers.environmentVariable("DWARF_DSYM_FOLDER_PATH")
        .orElse(layout.buildDirectory.dir("dsyms").map { it.asFile.absolutePath })
    into(dest)

    doLast { logger.lifecycle("collected Shared.framework.dSYM → ${dest.get()}") }
}

tasks.named("linkReleaseFrameworkIosArm64") { finalizedBy(collectFrameworkDsym) }

#!/usr/bin/env bash
# tools/upload-ios-symbols.sh
#
# Uploads every dSYM in the archive and ASSERTS that the shared framework's is among them.
# The previous step uploaded whatever `find` returned, which was correct behaviour over an
# incomplete directory — so the fix is not a better find, it is an assertion about what must
# be there.
set -euo pipefail
ARCHIVE="$1"   # preflight-ignore: a positional arg, not a secret
PLIST="${2:-iosApp/GoogleService-Info.plist}"   # preflight-ignore: a path, not a secret

APP="$ARCHIVE/Products/Applications/iosApp.app"   # preflight-ignore: a path
FRAMEWORK_BIN="$APP/Frameworks/Shared.framework/Shared"   # preflight-ignore: a path
[ -f "$FRAMEWORK_BIN" ] || { echo "FATAL: no Shared framework in the archive" >&2; exit 1; }

# The UUID the crash reports will reference. Read from the SHIPPED binary, not from the build
# directory — a build directory can hold a dSYM from a previous link, and uploading that one
# matches nothing while reporting success.
want=$(dwarfdump --uuid "$FRAMEWORK_BIN" | awk '{print $2}' | tr -d '-' | tr 'a-f' 'A-F')
echo "shipped framework UUID: $want"

found=""
while IFS= read -r -d '' dsym; do
  have=$(dwarfdump --uuid "$dsym" 2>/dev/null | awk '{print $2}' | tr -d '-' | tr 'a-f' 'A-F')
  echo "  uploading $(basename "$dsym")  [$have]"
  "$FIREBASE_PATH/upload-symbols" -gsp "$PLIST" -p ios "$dsym"
  # `if`, not `[ ... ] && ...` — under `set -e` the AND-list form aborts the loop on the
  # first non-match, and the first dSYM iterated is the app's, which never matches.
  if [ "$have" = "$want" ]; then found="$dsym"; fi
done < <(find "$ARCHIVE/dSYMs" -name '*.dSYM' -print0)

# The assertion. Without it this script is the old one: it uploads what it finds and reports
# success, and "what it finds" was missing the only dSYM that mattered for a year.
if [ -z "$found" ]; then
  cat >&2 <<EOF
FATAL: no uploaded dSYM matches the shipped Shared framework ($want).

  Every Kotlin frame in every crash report from this build will be a hex address, and
  nothing will fail — Crashlytics lists the UUID under "Missing dSYMs" and the upload
  step exits zero.

  The framework is linked by GRADLE, not Xcode, so its dSYM is not in the archive unless
  :shared:collectFrameworkDsym ran. Check that the link task's finalizedBy is wired.
EOF
  exit 1
fi
echo "shared framework symbols uploaded and UUID-matched"

And the output it produced, on the release that had been shipping unsymbolicated:

$ bash tools/upload-ios-symbols.sh build/iosApp.xcarchive

shipped framework UUID: 4F2A1C9E7B3D3E219A041C8E5B2D7F60
  uploading iosApp.app.dSYM  [B4D0E71AC3F24A8E9D015E7B3A2C8F41]
FATAL: no uploaded dSYM matches the shipped Shared framework (4F2A1C9E…).
  …

$ # after wiring collectFrameworkDsym
$ ./gradlew :shared:linkReleaseFrameworkIosArm64
collected Shared.framework.dSYM → /…/iosApp.xcarchive/dSYMs
$ bash tools/upload-ios-symbols.sh build/iosApp.xcarchive
  uploading iosApp.app.dSYM           [B4D0E71A…]
  uploading Shared.framework.dSYM     [4F2A1C9E…]
shared framework symbols uploaded and UUID-matched

And the same crash, from the next release:

Fatal Exception: KotlinNullPointerException
  0  Shared   FeedParser.parse(FeedParser.kt:14)
  1  Shared   FeedRepository$loadFeed$1.invokeSuspend(FeedRepository.kt:28)
  2  Shared   ContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  …

How we proved it#

That the framework's dSYM was never in the archive — which is why the find-based step was correct and useless:

$ ls "$ARCHIVE/dSYMs"
iosApp.app.dSYM
$ find "$ARCHIVE/dSYMs" -name '*.dSYM' | wc -l
1
$ find shared/build -name '*.dSYM' -maxdepth 6
shared/build/bin/iosArm64/releaseFramework/Shared.framework.dSYM

Produced by every release build, in a directory the archive step does not read.

That the UUID changes per link. The measurement that made a manual upload untenable:

  link #   source changed?   dSYM UUID
    1            —           4F2A1C9E-…
    2            no          B9041EE7-…
    3            no          C72E8A04-…

Three links, identical source, three UUIDs. A stored dSYM matches one build.

That sourceInfoType fixes the other thing. Both measured, because the setting is genuinely useful and for runtime traces:

                                   e.stackTraceToString()   crash report frames
  default                          addresses                addresses
  sourceInfoType=libbacktrace      file:line                addresses      ← unchanged
  + the dSYM uploaded              file:line                file:line

That the failure was silent on both sides. Established rather than assumed, because a year is a long time for nobody to notice:

  signal                                        present?
  upload-symbols exit code                      0
  CI step failure                               none
  Crashlytics API error                         none
  Crashlytics dashboard "Missing dSYMs"         yes — 1 entry, a bare UUID
  anything naming "Shared" or "Kotlin"          no

The only signal was a hexadecimal string in a dashboard section, with no indication which artifact it belonged to.

That the gate catches a broken wiring. The fixture is the realistic regression — somebody removes the finalizedBy while refactoring the build:

$ ./gradlew :shared:linkReleaseFrameworkIosArm64 -x :shared:collectFrameworkDsym
$ bash tools/upload-ios-symbols.sh build/iosApp.xcarchive
FATAL: no uploaded dSYM matches the shipped Shared framework (C72E8A04…).
  … Check that the link task's finalizedBy is wired.
exit 1

That reading the UUID from the shipped binary matters. The subtler trap, and the reason the script does not read the build directory:

  source of the expected UUID        result when build/ holds a stale dSYM
  shared/build/.../Shared.dSYM       matches the stale one → uploads it → reports SUCCESS
  the archive's embedded binary      mismatch → FATAL

A build directory can hold a dSYM from a previous link, so comparing the upload against the build directory would have passed while uploading symbols for a binary that was never shipped.

What I cannot tell you is how many of the past year's crashes could now be read. Crashlytics symbolicates at ingest, so historical reports stay as addresses — the dSYMs for those builds are gone, and the only recoverable information is the crash count per issue. Roughly 14,000 Kotlin crashes over the year are permanently unreadable.

And an honest gap. The simulator framework has its own UUID and its own dSYM, and we do not upload it — debug and simulator crashes are not in Crashlytics. That is deliberate, and it means a TestFlight build linked against the simulator slice would be unsymbolicated, which has not happened and is not prevented.

What this generalizes to#

An artifact built by one toolchain and consumed by another falls into the gap between their conventions, and the collection step on each side is correct about its own half. The dangerous version is where the consuming system fails silently — a crash report with unresolved frames is still a valid crash report — so nothing accumulates pressure to fix it. The reliable form is to assert the thing you need rather than to collect whatever is there: our old step was a correct find over an incomplete directory, and no improvement to the find would have helped.

---

Related#

• Four crash reporters, one shared module — the per-target artifact table, and the grouping key that made the volume visible.
• The shared module cost 400KB on Android and 11MB on iOS — the other consequence of the framework being a separate statically-linked binary.

Keep reading#

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.