The retry that turned a blip into an outage

TL;DR#

A deploy caused eleven seconds of 503s. Every client retried with exponential backoff, which is the recommended thing to do, and the eleven-second blip became a forty-minute outage.

The reason is that exponential backoff without jitter is synchronising. Every client failed at roughly the same moment, so every client retried at the same moment — 1s, 2s, 4s, 8s — arriving as increasingly large simultaneous waves. Each wave knocked the recovering service back down.

Backoff spaces out one client's retries. Jitter is what spaces out the fleet's. The second one is the property that decides whether your retry policy helps or amplifies.

The case this post is about#

A mobile client syncing on app open, with the retry policy everyone writes:

suspend fun <T> withRetry(attempts: Int = 5, block: suspend () -> T): T {
    var delayMs = 1_000L
    repeat(attempts - 1) {
        try { return block() } catch (e: RetryableException) { }
        delay(delayMs)
        delayMs *= 2            // 1s, 2s, 4s, 8s — textbook exponential backoff
    }
    return block()
}

Reviewed, approved, and matching every guide. About 90,000 clients were using it.

The symptom#

The blip and what followed:

14:02:11  deploy rollout, 11s of 503s from one instance group
14:02:22  instances healthy, 503s stop
14:02:23  request rate 4× baseline      ← retry wave 1
14:02:25  request rate 7× baseline      ← wave 2 (the 1s cohort now on its 2s delay)
14:02:29  request rate 11× baseline     ← wave 3
14:02:37  request rate 14× baseline     ← wave 4
14:02:38  connection pool exhausted, 503s resume
…
14:41:50  recovered, after shedding traffic at the edge

Three things in that timeline are the whole story.

The service was healthy at 14:02:22. Everything after that was our own traffic.

The waves grew. Not decayed — each wave was larger than the last, because clients that had been failing since the start were joining later waves while new clients began their first attempts.

And the retries had no idea the service had recovered, because every one of them was scheduled before it did.

Why the obvious fix failed#

"Reduce the retry count from 5 to 3." Reduces the number of waves and not their synchronisation. We did this first and the next incident was shorter and still an incident.

"Increase the base delay to 5 seconds." Moves the waves later and makes them bigger, because more clients accumulate into each one. Measurably worse in the load test.

"Add a circuit breaker per client." Genuinely useful, and a per-client breaker opens only after that client has failed several times — so the first two waves happen anyway. It limits the tail, not the onset.

"Rate-limit at the edge." This is what ended the incident, and as a design it means accepting that your clients will attack you and building a wall. Worth having; not a reason to leave the clients misbehaving.

The actual mechanism#

Two things that sound like the same thing and are not.

Backoff spreads one client's attempts over time. It protects the service from a single client hammering it, and it gives a recovering dependency time between that client's requests.

Jitter spreads the fleet's attempts over clients. Without it, every client that failed at the same instant retries at the same instant — because they all compute the same delay from the same starting point.

A shared failure is a synchronising event. 90,000 clients hit a 503 within the same few hundred milliseconds, so 90,000 timers were set for +1s. Deterministic backoff then keeps them in lockstep for every subsequent attempt.

The arithmetic is unforgiving. If the sync window is 200ms and baseline is 300 requests/second, then 90,000 retries landing in a 200ms window is an instantaneous rate of 450,000/second — 1,500× baseline, compressed into a moment, aimed at a service whose connection pool is sized for 300.

Backoff is about one client's politeness. Jitter is about the fleet's correlation. A failure that hits everyone at once correlates everyone, so a policy without jitter converts a brief outage into a self-sustaining one.

And one more property that made recovery impossible: retries were counted as load but not as failures the service could signal about. There was no way for the server to say "stop retrying" — which is what Retry-After exists for and what nobody was reading.

There is also a layering effect that turns one client's reasonable policy into an unreasonable one, and it is invisible from inside any single component. Three retries at each of three layers is not nine attempts spread politely — it is 27, and the innermost layer sees all of them:

mobile client      3 attempts
  → API gateway    3 attempts each        =  9
    → service      3 attempts each        = 27 requests at the database

Each layer's author made the same defensible choice, nobody specified more than three, and the database experiences a 27× amplification of a failure it is already struggling with. Worse, the timeouts compound in the same direction: if each layer waits for its own retries before reporting failure, the user's request is still outstanding long after the client has given up — so the work being retried is work nobody is waiting for any more.

The rule that falls out is that retry belongs at exactly one layer, and every other layer fails fast and propagates. Which layer is a design decision — usually the one closest to the user, because it is the only one that knows whether anybody is still there.

The fix#

Four changes, in order of how much they mattered.

Full jitter, which is the actual fix:

suspend fun <T> withRetry(
    attempts: Int = 4,
    baseMs: Long = 1_000,
    maxMs: Long = 30_000,
    random: Random = Random,
    block: suspend () -> T,
): T {
    repeat(attempts - 1) { attempt ->
        try { return block() } catch (e: RetryableException) {
            if (e.retryAfterMs != null) { delay(e.retryAfterMs); return@repeat }  // obey the server
            val ceiling = min(maxMs, baseMs shl attempt)
            delay(random.nextLong(0, ceiling + 1))        // FULL jitter: uniform in [0, ceiling]
        }
    }
    return block()
}

Full jitter — uniform in [0, ceiling] rather than ceiling ± something — is the variant worth defaulting to. "Equal jitter" (ceiling/2 + random(ceiling/2)) halves the correlation; full jitter removes most of it, at the cost of some retries happening sooner than a strict backoff would like. For protecting a shared dependency, that trade is correct.

Obey Retry-After. The server knows things the client does not:

HTTP/1.1 503 Service Unavailable
Retry-After: 30

A client that reads this stops guessing. Adding it server-side during overload is how you coordinate a fleet you do not control.

A retry budget, so retries cannot exceed a fraction of real traffic:

// per client: refuse to retry if retries have been >10% of requests recently
class RetryBudget(private val ratio: Double = 0.1) {
    fun tryConsume(): Boolean = retries.rate() < requests.rate() * ratio
}

This is the backstop that bounds amplification independently of timing. Even with every timing assumption wrong, retries cannot become more than a tenth of load.

And deduplicate at the source. A large share of the retries were for the same sync from the same client, triggered by several code paths. Collapsing them cut the baseline retry volume by about half before any of the above applied.

How we proved it#

Replaying the incident in a load test, because arguing about it was not converging:

90,000 simulated clients, 11s of injected 503s, service pool sized for 300 rps

                          peak rps   time to recover
deterministic backoff      418,000    38m 20s        ← reproduced the incident
backoff + equal jitter      47,000     6m 10s
backoff + full jitter        3,900        14s        ← recovered with the blip
full jitter + budget         3,100        12s

The first row reproducing the real incident is what made the rest trustworthy. The jump from 418,000 to 3,900 is one Random.nextLong call.

That jitter is what did it, isolated. Same test, holding everything else fixed:

attempts=5, base=1s, deterministic   418,000 peak
attempts=5, base=1s, full jitter       3,900 peak
attempts=3, base=1s, deterministic   211,000 peak    ← fewer retries, still an outage
attempts=5, base=5s, deterministic   509,000 peak    ← bigger delay, WORSE

Those last two rows are the ones I would want a reviewer to see, because they are the two fixes everyone proposes first, and one of them is actively harmful.

And a unit test, so the jitter cannot be removed by a future tidy-up:

@Test
fun retryDelaysAreSpreadAcrossClients() = runTest {
    val delays = (1..1_000).map { seed ->
        var observed = 0L
        val r = Random(seed)
        runCatching { withRetry(attempts = 2, random = r) { throw Retryable() } }
        observed = currentTime
        observed
    }
    // with deterministic backoff every element is identical
    assertTrue(delays.distinct().size > 900, "delays not spread: ${delays.distinct().size}")
}

without jitter:  FAILED — delays not spread: 1
with jitter:     PASSED

Two honest limits. The load test used a uniform client population and one failure shape; a real fleet has heterogeneous clients and correlated-but-not-identical failure times, so the 418,000 is an upper-bound reconstruction rather than a measurement of the incident itself. And the edge rate-limit stayed in place — the client fix reduces the need for it, and removing a working backstop because the clients are now polite would be the wrong lesson.

What this generalizes to#

A shared failure correlates every client, and correlation is the thing that hurts. Cache expiry at a fixed TTL, cron jobs at midnight, tokens with identical lifetimes, a deploy restarting every pod at once. Anywhere many actors compute the same time from the same event, you have built a wave. Jitter everything with a schedule.

Politeness per actor is not politeness in aggregate. Each client's behaviour was reasonable and well-reviewed. The emergent behaviour of 90,000 of them was an attack. Reviewing a retry policy means asking what happens when every instance of it runs simultaneously.

Let the server tell the client what to do. Retry-After, backpressure signals, a "slow down" response. Clients guessing is strictly worse than clients being told, and the signalling path is usually a few lines on both sides.

A budget bounds what timing cannot. Timing fixes depend on assumptions about failure distribution. A ratio cap — retries may never exceed N% of traffic — holds even when every assumption is wrong, which makes it the right thing to have alongside the timing fix rather than instead of it.

Related#

• 429 is not an error, it is a schedule — when retrying IS the right answer
• The third time this filled up, I diagnosed it as an outage — retries against a failure that cannot clear
• One Ktor client, four engines, and the timeouts that don't mean the same thing — why retry does not cover hangs

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.