Every timeout you set independently adds up to one that is wrong

TL;DR#

Each service in a call chain sets a sensible timeout. The client waits 30s, the gateway waits 30s on the API, the API waits 30s on the database. Every number is defensible in isolation, and the user's worst case is 90 seconds — or more, once retries multiply it.

Timeouts do not compose, because a timeout is a duration and what you need is a deadline: an absolute instant, set once at the edge, propagated down, and respected by every hop. Then a hop that receives 200ms of remaining budget does not start a 30-second query.

Deadlines also give you something a duration cannot: the ability to refuse work that cannot possibly finish in time.

The case this post is about#

A checkout request through four hops, each with its own timeout:

// mobile client
val http = HttpClient { install(HttpTimeout) { requestTimeoutMillis = 30_000 } }

// gateway → order service
val orderClient = HttpClient { install(HttpTimeout) { requestTimeoutMillis = 30_000 } }

// order service → pricing service
val pricingClient = HttpClient { install(HttpTimeout) { requestTimeoutMillis = 20_000 } }

// pricing service → database
val query = db.query(sql, timeout = 15.seconds)

Four numbers, each chosen by someone who thought about it. Nobody chose the total.

The symptom#

A user-visible wait of 94 seconds. From a trace of one slow checkout:

t=0      client → gateway
t=0      gateway → order
t=0      order → pricing
t=0      pricing → db          (db is degraded, queries taking ~14s)
t=14     pricing retries db    (its own retry policy)
t=28     pricing retries db
t=42     pricing gives up, 503 to order
t=42     order retries pricing
t=62     order gives up (20s timeout), 503 to gateway
t=62     gateway retries order
t=94     client times out at 30s... but the client had already retried twice

The client gave up long before anything else did. At t=30 the client timed out and the user saw an error — while four services carried on working on a request nobody was waiting for:

work performed after the client disconnected:  64 seconds across 3 services

And the retries multiplied the budget. Each layer's retry count multiplies the layer below: 2 × 2 × 3 = 12 database queries for one checkout attempt.

The database never recovered, because the load was self-sustaining — work for abandoned requests kept the degradation going.

Why the obvious fix failed#

"Make the inner timeouts shorter than the outer ones." The right direction and still wrong, as we discovered by doing it. We set db=5s, pricing=10s, order=15s, client=30s. It bounds the total, and every hop still spends its full allowance regardless of how much time the request has actually used — so a request that burnt 14s on an earlier step still starts a fresh 5-second query.

"Lower everything." Faster failures and more of them: legitimate slow requests started failing. Tightening a static ladder trades one error class for another.

"Remove the inner retries." Reduces the multiplication and loses genuinely useful retries on transient blips. Retries are not the problem; unbudgeted retries are.

"Just set the client timeout and let the rest run." This is the status quo, and it is what produced 64 seconds of orphaned work per failed checkout.

The actual mechanism#

A timeout is a relative duration measured from when a hop starts waiting. A deadline is an absolute instant by which the whole operation must finish.

The difference is everything:

Durations do not compose. Hop N's 30 seconds is independent of how long hops 1..N-1 took. The total is the sum along the longest path, times the retry multipliers.

There is a second-order cost that follows from the first and is easy to miss: work continues after nobody is waiting for it. Once the client at the edge has given up at 10 seconds, every hop below it is still holding a connection, a thread and a database transaction for a response that will be discarded on arrival.

So the system is at its busiest precisely when it is least useful — the capacity is occupied by requests whose results cannot be delivered, while new requests queue behind them. That is the mechanism that turns a slow dependency into an outage rather than a slowdown, and no per-hop number can fix it, because a duration carries no information about whether the caller is still there.

A deadline does. Passing an absolute instant downward means every hop can ask the one question that matters before starting expensive work — is there still time? — and a hop that answers no returns immediately instead of consuming capacity on behalf of someone who left.

Deadlines compose by construction. Set deadline = now + 10s at the edge, pass it down, and every hop computes its own allowance as deadline - now. The total is 10 seconds by definition, regardless of path length.

And deadlines enable two things durations cannot:

Fail fast on arrival. A hop receiving 50ms of remaining budget for an operation that takes 200ms can refuse immediately rather than starting work that is guaranteed to be wasted.

Bounded retries. A retry only happens if the remaining budget allows it, so retry multiplication is impossible — the budget is the ceiling.

A timeout answers "how long will I wait here". A deadline answers "when does the user stop caring". Only the second one is a property of the request, and only the second one composes.

The fix#

One deadline at the edge, propagated as a header, enforced at every hop.

// edge — the ONE place a budget is chosen, from the user-facing SLO
val deadline = clock.now() + 10.seconds

// propagation — an absolute instant, so clock-relative drift is explicit
request.header("X-Request-Deadline", deadline.toEpochMilliseconds().toString())

// commonly used helper — every outbound call derives its timeout from the budget
class Deadline(private val at: Instant, private val clock: Clock) {

    fun remaining(): Duration = (at - clock.now()).coerceAtLeast(Duration.ZERO)

    /** Refuse work that cannot finish. The caller gets a fast, honest failure. */
    fun requireAtLeast(needed: Duration) {
        val left = remaining()
        if (left < needed) throw DeadlineExceeded(needed = needed, remaining = left)
    }
}

// pricing service → database
suspend fun price(cart: Cart, deadline: Deadline): Price {
    deadline.requireAtLeast(200.milliseconds)        // a query under this is pointless
    return withTimeout(deadline.remaining()) {        // not a constant — the budget
        db.query(sql)
    }
}

// retries are budget-bounded, so they cannot multiply
suspend fun <T> retryWithinBudget(deadline: Deadline, block: suspend () -> T): T {
    var attempt = 0
    while (true) {
        try { return block() } catch (e: RetryableException) {
            attempt++
            val backoff = Random.nextLong(0, min(1000L shl attempt, 4_000L)).milliseconds
            if (deadline.remaining() < backoff + MIN_ATTEMPT) throw e   // no time left: give up
            delay(backoff)
        }
    }
}

Five decisions:

One budget, chosen from the product SLO. Ten seconds because that is when a user abandons checkout — not because it is a round number a service picked.

An absolute instant on the wire, not a remaining duration. A duration has to be decremented by each hop and loses the time spent in transit. An instant needs only roughly-synchronised clocks, which NTP gives you, and the skew is a known small quantity rather than accumulating error.

requireAtLeast refuses doomed work. This is what eliminated the orphaned load: a hop with 80ms left does not start a 200ms query, so the database stops receiving requests for abandoned checkouts.

Retries check the budget before sleeping. The multiplication is structurally impossible.

And clock skew is handled by clamping, not trusted:

// refuse a deadline implausibly far in the future — a skewed or hostile caller
val cap = clock.now() + MAX_BUDGET          // 30s
val effective = minOf(parsedDeadline, cap)

Plus server-side cancellation, so a disconnect stops the work:

// the request scope is cancelled when the client disconnects;
// every suspending call below it is cooperative, so the work actually stops

How we proved it#

The worst case, measured against a degraded dependency. A load test with the database artificially slowed to 14-second queries:

                              p99 user-visible latency   db queries per checkout
independent timeouts (orig)            94.2s                      12
static ladder (5/10/15/30)             31.8s                       6
deadline propagation (10s)             10.1s                       1

Twelve queries to one is the thing that let the database recover. The middle row is the fix that looked sufficient.

Orphaned work, which was the load-amplifying part:

work performed after client disconnect, per failed checkout
  before:  64.1s across 3 services
  after:    0.3s                       ← cancellation + requireAtLeast

That legitimate slow requests still succeed — the thing "lower everything" broke:

checkouts taking 6–9s (slow but valid), success rate
  independent timeouts:   99.1%
  static ladder (tight):  91.4%      ← broke valid requests
  deadline (10s):         99.3%

That row is why the budget is 10 seconds and not 5: the deadline bounds the total without truncating work that was going to finish.

And a test that the budget cannot be exceeded regardless of path depth:

@Test
fun totalLatencyIsBoundedByTheBudgetNotByPathLength() = runTest {
    val deadline = Deadline(at = testClock.now() + 10.seconds, clock = testClock)
    val result = runCatching { fourHopChain(slowAtEveryHop = true, deadline) }
    assertTrue(result.isFailure)
    assertTrue(currentTime <= 10_200, "took ${currentTime}ms")   // 10s + slack
}

with independent timeouts:  FAILED — took 94,180ms
with deadline propagation:  PASSED — took 10,060ms

Two honest limits. Deadline propagation requires every hop to participate; one service that ignores the header reintroduces its own unbudgeted wait, and we found two such services (a third-party gateway and an internal batch endpoint) that we had to bound with a static timeout instead. And the approach assumes clocks within a second or so of each other — true with NTP, and the MAX_BUDGET clamp exists because it is an assumption rather than a guarantee.

What this generalizes to#

Durations do not compose; deadlines do. This is the whole post, and it applies to anything with a call chain — RPC, HTTP, job pipelines, nested transactions. Per-hop durations sum along the path and multiply with retries; one absolute deadline is invariant to path shape.

Derive the budget from the user, not from the service. The only principled source for a timeout is when the person stops caring. Everything below that is an allocation of that budget, and services inventing their own numbers is how 30+30+30 happens.

Refusing doomed work is a feature. A hop that knows it cannot finish should fail immediately rather than consuming capacity. This is what stops a degraded dependency from being kept degraded by requests nobody is waiting for.

Bound retries by budget, not by count. A retry count multiplies down the stack. A budget check before each attempt makes the total bounded regardless of how many layers decided to be resilient.

Related#

• The retry that turned a blip into an outage — the retries this budget bounds
• One Ktor client, four engines, and the timeouts that don't mean the same thing — what a single timeout actually enforces
• The lock held across a network call, and the deadlock that followed — unbounded waits in another costume

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.