Nothing reconnects your WebSocket, on any target

TL;DR#

• A chat socket that "worked" for weeks silently stopped delivering messages whenever the network blipped. No exception, no error callback, no log line — the incoming channel simply closed and our for (frame in incoming) loop ran to completion like a successful read.
• Ktor's WebSocketSession gives you a session, not a connection manager. Nothing in Ktor, OkHttp, Darwin or the JS engine reconnects for you, and two of the four engines close the channel without throwing.
• Reconnection is application code: a loop, bounded jitter, and a resume token so the gap is refilled rather than skipped. All four targets run the identical implementation.
• What reading on buys: the per-engine termination behaviour measured, why isActive is the wrong predicate, and the missed-message count that a resume token recovered.

The case this post is about#

A chat feed over Ktor's WebSocket plugin, written as a straight read loop:

// core/network/src/commonMain/kotlin/ChatSocket.kt
class ChatSocket(private val client: HttpClient) {

    fun messages(room: String): Flow<ChatMessage> = flow {
        client.webSocket("wss://api.example.com/rooms/$room") {
            // Reads until the channel closes. Which it does, quietly, on every network blip.
            for (frame in incoming) {
                if (frame is Frame.Text) emit(Json.decodeFromString(frame.readText()))
            }
        }
    }
}

// feature/chat
init {
    scope.launch {
        socket.messages(roomId).collect { messages.update { it + listOf(message) } }
    }
}

The symptom#

Users reporting a chat that stopped updating until the app was restarted:

"Messages stop arriving. If I force-quit and reopen, everything's there."

Which is a precise bug report: the messages existed server-side and the client had stopped receiving. And the client logs at that moment held nothing:

# 14 reproductions, instrumented
  exceptions thrown from messages()                 0
  error logs                                        0
  "socket closed" log lines                         0
  collect { } invocations after the drop            0
  Flow completions (onCompletion without error)    14   ← it completed SUCCESSFULLY

Fourteen for fourteen, the flow completed normally. From the collector's point of view a Flow<ChatMessage> that completes is a stream that has finished — indistinguishable from a server that legitimately closed the room.

And the conditions, once we could reproduce it:

# drop rate per hour of foreground use, measured over 3 days / 41 sessions
  cause                              drops/hr
  wifi → cellular handover              2.1
  cellular cell handover                1.4
  server-side idle timeout (60s)        0.9   ← the one we were causing ourselves
  app backgrounded > 30s                — (socket killed by the OS)
  total                                 4.4

Four and a half drops per hour, each one permanent until an app restart.

Why the obvious fix failed#

The first fix was a try/catch with a retry:

fun messages(room: String): Flow<ChatMessage> = flow {
    while (true) {
        try {
            client.webSocket(…) { for (frame in incoming) { … } }
        } catch (e: Exception) {
            delay(1000)       // retry
        }
    }
}

Which never retried on Android, because on the OkHttp engine the drop does not throw — the incoming channel closes and the for loop exits normally, falling through to while (true) and reconnecting immediately with no delay. On iOS it threw, caught, and waited a second. Same code, two behaviours, and the Android path turned a network blip into a reconnect storm:

  engine                      on a dropped socket        the loop did
  OkHttp (android)            channel closes, no throw   reconnect immediately, ~40×/s
  Darwin (ios)                throws                     catch, delay 1s, reconnect

The second fix was a fixed delay plus a check on the session:

while (session.isActive) { … }

isActive is the coroutine job's state, not the socket's. It is true for a session whose transport has gone away, which is why the predicate reads as correct and changes nothing.

The third attempt was a reconnect with exponential backoff and no resume token, which fixed the "stops forever" symptom and left a quieter one: every reconnect skipped whatever was sent during the gap.

# messages sent server-side during the reconnect window, 3 days
  reconnects                       412
  reconnects with a gap > 0        180
  messages never delivered         1,104

A chat that silently drops 1,104 messages is worse than one that visibly stops, because the user cannot tell.

The actual mechanism#

Three facts, and the third is the one that costs money.

A WebSocket close is a normal event, not an error. The protocol has a close handshake with a status code; an orderly close is a successful end-of-stream. Ktor models incoming as a ReceiveChannel<Frame>, and a closed channel ends a for loop without throwing. That is correct channel semantics and it means the happy path and the failure path are the same code path.

Whether a drop throws is an engine decision. Measured rather than read:

  engine            abrupt transport loss     server close frame     idle timeout
  OkHttp (JVM)      channel closes            channel closes         channel closes
  Darwin (iOS)      throws                    channel closes         throws
  Js (browser)      channel closes            channel closes         channel closes
  CIO (desktop)     throws                    channel closes         throws

Two engines throw on some drops. None throws on all of them. So an exception handler cannot be the detection mechanism — the only reliable signal is "the loop ended", and that signal is identical for a healthy close.

And a reconnect without a resume token is a data gap. This is the part that is not about Ktor at all: the server advanced while you were away. Unless you tell it where you stopped, it resumes from now, and the messages in between are gone for that client. The server has to support this — ours exposed ?since=<messageId> and we had never used it, because with no reconnect loop there had never been a gap to fill.

  reconnect strategy                      gap handling
  immediate reconnect                     messages during the gap lost
  backoff + jitter                        messages during the gap lost (just fewer gaps)
  backoff + jitter + resume token         gap refilled on reconnect

The fix#

One reconnect loop in shared code, detection on loop-exit rather than on exception, bounded full-jitter backoff, and a resume token threaded through so the gap is refilled.

// core/network/src/commonMain/kotlin/ChatSocket.kt
class ChatSocket(private val client: HttpClient, private val clock: Clock) {

    fun messages(room: String): Flow<ChatEvent> = flow {
        var lastSeen: String? = null
        var attempt = 0
        var closeReasonSeen: CloseReason? = null

        while (currentCoroutineContext().isActive) {
            // Detection is LOOP EXIT, not an exception: two of four engines close the channel
            // without throwing, and no engine throws on every drop. A catch-based design works
            // on iOS and reconnect-storms on Android — measured at ~40 reconnects/second.
            val closedCleanly = runCatching {
                client.webSocket(
                    urlString = buildString {
                        append("wss://api.example.com/rooms/$room")
                        // The resume token. Without it a reconnect resumes from NOW and the
                        // gap is lost: 1,104 messages across 180 gapped reconnects in 3 days.
                        lastSeen?.let { append("?since=$it") }
                    },
                ) {
                    attempt = 0                       // reset only after a connection is OPEN
                    emit(ChatEvent.Connected)
                    for (frame in incoming) {
                        if (frame !is Frame.Text) continue
                        val msg = Json.decodeFromString<ChatMessage>(frame.readText())
                        // EMIT first, advance second. The other order loses the message when
                        // the emit is cancelled: the token would say it was delivered when it
                        // was not. This way a cancellation mid-emit re-delivers it after the
                        // resume — at-least-once, which costs a possible duplicate and never
                        // a hole. The duplicate is the collector's to drop.
                        emit(ChatEvent.Message(msg))
                        lastSeen = msg.id
                    }
                    // Reached only when the channel closed. Ask the session WHY, and KEEP the
                    // answer — it is read outside this block, after runCatching unwinds.
                    closeReasonSeen = closeReason.await()
                }
            }.isSuccess

            // A deliberate server-side close is a terminal state, not something to retry.
            // Retrying these produced a reconnect every 300 ms against a server that kept
            // saying no.
            val terminal = setOf(CloseReason.Codes.NORMAL, CloseReason.Codes.GOING_AWAY)
            if (closedCleanly && closeReasonSeen?.knownReason in terminal) {
                emit(ChatEvent.Ended); return@flow
            }
            closeReasonSeen = null

            emit(ChatEvent.Reconnecting(attempt))

            // FULL jitter, not a fixed ladder: every client on a cell that drops shares the
            // same failure instant, so a deterministic backoff reconnects them in lockstep.
            // random(0, cap) spreads the fleet; backoff alone only spreads one client.
            val cap = minOf(30_000L, 500L * (1L shl minOf(attempt, 6)))
            delay(Random.nextLong(0, cap.coerceAtLeast(1)))
            attempt++
        }
    }
}

// The consumer now distinguishes "finished" from "lost the connection" — which the original
// Flow<ChatMessage> could not express, since both arrived as a normal completion.
sealed interface ChatEvent {
    data object Connected : ChatEvent
    data class Message(val msg: ChatMessage) : ChatEvent
    data class Reconnecting(val attempt: Int) : ChatEvent
    data object Ended : ChatEvent
}

And the output it produced, same three days of real use:

$ ./tools/socket-report.sh --days 3

                                    original   backoff,     shipped
                                                 no token
  drops                                  412        412          408
  permanent stalls (until app restart)   412          0            0
  reconnects                               0        412          408
  messages lost                          all*       1,104           0
  median time to reconnect                 —        1.4 s       340 ms
  p99 time to reconnect                    —       11.8 s        4.2 s
  reconnect storms (>5/s)                 —           37            0

  * every message after the drop, until the user restarted the app — there was no gap to
    measure because the stream never resumed.

$ # the engine-behaviour test, which is what makes one implementation safe on four targets
$ ./gradlew :core:network:allTests --tests '*SocketReconnect*'
reconnectsWhenTheChannelClosesWithoutThrowing   PASSED × 4 targets
reconnectsWhenTheEngineThrows                   PASSED × 4 targets
doesNotReconnectOnNormalClose                   PASSED × 4 targets
resumesFromTheLastSeenId                        PASSED × 4 targets

How we proved it#

That the drop does not throw on two engines — the measurement that invalidated the try/catch design, and the reason one implementation had two behaviours:

$ # kill the socket at the OS level mid-stream, per target
  android (OkHttp)        for-loop exited; runCatching.isSuccess == true
  ios (Darwin)            SocketException; isSuccess == false
  desktop (CIO)           SocketException; isSuccess == false
  web (Js)                for-loop exited; isSuccess == true

That the flow completed successfully. The signal that made the original bug invisible, confirmed rather than inferred:

socket.messages(room)
    .onCompletion { e -> println("completed, cause=$e") }
    .collect { }
// completed, cause=null         ← 14 of 14 reproductions

cause=null is what a legitimately-finished stream looks like. There was nothing to log.

That isActive is the wrong predicate. Checked because it is the obvious guard and it reads as correct:

  state                                 session.isActive
  transport alive                             true
  transport gone, job not cancelled           true   ← here
  job cancelled                               false

That full jitter matters more than the backoff curve. Measured with 2,000 simulated clients sharing one drop instant, because a fixed ladder is the usual implementation:

  policy                          peak reconnects/s     server 503s
  immediate                              2,000              yes
  fixed 1s                               1,980              yes   ← lockstep; the delay does nothing
  exponential, no jitter                 1,970              yes
  exponential + full jitter                 71              no

A shared failure instant synchronises everyone, so a deterministic delay moves the spike rather than flattening it.

That the resume token recovers the gap. The assertion in the test, and the production number:

$ ./gradlew :core:network:allTests --tests '*resumesFromTheLastSeenId*'
  sent during the gap:      5 messages
  delivered after resume:   5
  duplicates:               0     ← ?since= is exclusive; a server that is inclusive needs a dedupe

That not retrying a 1000 close is load-bearing. Found by shipping without the check:

  without the NORMAL-close check:  reconnect every ~300 ms against a server returning 1000
  with it:                          one connect, one Ended event

What I cannot tell you is whether 30 seconds is the right cap. It is long enough that a user who puts the phone down gets a reconnect on the first attempt after picking it up, and short enough not to look broken — and we have not tested the case where the server is down for minutes, where a 30-second ceiling means a reconnect attempt every half minute per client indefinitely. A retry budget (a cap on total attempts per window) is the missing control.

And an honest gap. lastSeen lives in the flow's local state, so a new collector starts with no token and resumes from now. That is correct for a fresh screen and wrong after a process death, where the user has a gap the socket layer cannot see. Persisting the token is a storage decision we have not made.

What this generalizes to#

When a transport models disconnection as a normal end-of-stream — a closed channel, an EOF, a completed iterator — the success path and the failure path become the same code, and an error handler cannot tell them apart. The reliable signal is the loop ending; the question to ask is "why did it end", which usually means reading a close reason rather than catching something. And once you reconnect, the second question arrives immediately and is not about the transport at all: the other side advanced while you were gone, and either you tell it where you stopped or you quietly lose whatever happened in between.

---

Related#

• The retry that made it worse — the jitter arithmetic above, from the incident where the clients took the server down.
• One Ktor client, four engines — the per-engine differences this post measures, catalogued.

Keep reading#

One post a week on shipping mobile software. No digests, no course pitch. Subscribe.