Idempotency is a product decision before it is a technical one

TL;DR#

"Make the endpoint idempotent" sounds like a technical task: add an Idempotency-Key header, store it, return the cached response on a repeat. That part takes an afternoon.

The hard part is deciding what counts as the same request — and that is a product question. Is a second £25 charge from the same user, same card, same cart, 90 seconds later a retry to deduplicate or a genuine second purchase? No amount of infrastructure answers that.

Get the identity definition wrong and you either double-charge people or silently refuse real purchases. Both are worse than the retry you were protecting against.

The case this post is about#

A payments endpoint, with the standard fix applied after a double-charge incident:

@Post("/charges")
suspend fun charge(@Header("Idempotency-Key") key: String, @Body req: ChargeRequest): Charge {
    idempotency.get(key)?.let { return it }           // replay
    val charge = gateway.charge(req.amount, req.token)
    idempotency.put(key, charge, ttl = 24.hours)
    return charge
}

Correct mechanics. The client generates a UUID per checkout attempt and sends it on every retry of that attempt.

Then three separate bug reports arrived, and all three were about the definition, not the mechanism.

The symptom#

A customer was double-charged anyway. The client had been generating a fresh key on each retry:

14:02:01  POST /charges  key=a1b2…  → 200, charged £25     (response lost to a timeout)
14:02:31  POST /charges  key=c3d4…  → 200, charged £25     ← new key, new charge

The server was idempotent and the client was not using it. A key generated per attempt rather than per intent is not an idempotency key.

A customer could not buy the same thing twice. After the client was fixed to derive the key from the cart contents:

val key = sha256("${userId}:${cartHash}:${amount}")   // "deterministic, so retries match"

10:00  buy 1 coffee  → charged
10:05  buy 1 coffee  → 200, replayed the first charge, no coffee

Deterministic from content means a genuine repeat purchase is indistinguishable from a retry.

And a replay returned a stale response. A key reused after the underlying order had been cancelled returned the original success, so the client showed a confirmation for an order that no longer existed.

Three bugs: key-too-narrow, key-too-broad, and response-staleness. The mechanism was identical in all three.

Why the obvious fix failed#

"Generate the key on the server." Impossible — the whole point is to recognise a repeat of something, which requires the client to tell you it is the same intent. A server-generated key is new every time by construction.

"Hash the request body." This is the too-broad failure above. Two legitimate identical requests exist (buy another coffee), and content hashing cannot tell them from a retry.

"Add a timestamp to the hash." Now every retry has a different key, which is the too-narrow failure. Any time-based component defeats the purpose, and a rounded timestamp just makes the window arbitrary.

"Use a shorter TTL." We tried 5 minutes, which fixed the repeat-purchase case for coffee and broke retries for a client that was offline for ten. The TTL is a guess at how long a retry might take, and it is not the same thing as how long an intent lasts.

The actual mechanism#

Idempotency has three parts, and only the third is infrastructure.

An identity definition — what makes two requests "the same". This is a product decision. It has to distinguish "the client is retrying because it did not hear back" from "the user did the thing again". Nothing in the request can tell those apart; only the client's intent can, which is why the client must mint the key.

A key lifetime — how long that identity remains meaningful. Also a product decision. A charge intent might be minutes; a subscription signup might be days.

A store and a replay rule — the easy part. Persist key → outcome, return the stored outcome on a repeat.

The identity question has a clean answer once framed properly:

The key identifies a user intent, created at the moment the intent is formed, and reused for every attempt to fulfil that one intent. Tapping "Pay" creates an intent and a key. Every retry of that tap reuses it. Tapping "Pay" again — a new decision by the user — creates a new intent and a new key.

The key is created where the intent is created — the button tap — not where the request is sent and not from what the request contains. That single placement decision resolves both failure modes.

And the staleness problem is a fourth question people skip: what does a replay return? The original response is right for "did my charge go through"; it is wrong if the world has moved on.

The fix#

Mint the key at the intent, persist it with the intent, and make the replay explicit about what it is returning.

// client — the key is part of the PendingCharge, created when the user taps Pay
@Serializable
data class PendingCharge(
    val localId: String = newLocalId(),
    val idempotencyKey: String = newSecretToken(),   // once, per intent
    val amountMinor: Long,
    val cartId: String,
)

// stored locally BEFORE the first attempt, so a process death does not lose it
suspend fun pay(cart: Cart) {
    val pending = PendingCharge(amountMinor = cart.totalMinor, cartId = cart.id)
    pendingStore.put(pending)                        // durable
    submit(pending)                                  // retries reuse pending.idempotencyKey
}

Persisting the key before the first attempt is the detail that makes it work across an app restart. A key held only in memory is lost exactly when a retry is most needed.

// server — store the outcome, and record enough to detect misuse
@Post("/charges")
suspend fun charge(@Header("Idempotency-Key") key: String, @Body req: ChargeRequest): Response {
    val fingerprint = sha256("${req.userId}:${req.amountMinor}:${req.cartId}")

    idempotency.get(key)?.let { prior ->
        // SAME key, DIFFERENT request = a client bug. Refuse rather than guess.
        if (prior.fingerprint != fingerprint) {
            return Response(422, ApiError("IDEMPOTENCY_KEY_REUSED",
                "This key was used for a different request."))
        }
        return Response(200, prior.outcome, headers = mapOf("Idempotent-Replay" to "true"))
    }

    // reserve the key BEFORE calling the gateway, so concurrent retries serialise
    if (!idempotency.reserve(key, fingerprint)) {
        return Response(409, ApiError("IDEMPOTENT_REQUEST_IN_FLIGHT", "Retry shortly."))
    }

    val outcome = gateway.charge(req.amountMinor, req.token)
    idempotency.complete(key, outcome, ttl = 7.days)
    return Response(200, outcome)
}

Four decisions:

Reserve before calling the gateway. Two concurrent retries must not both reach the payment provider. A read-then-write check has a window; a conditional insert does not.

A fingerprint mismatch is a 422, not a replay. If the same key arrives with a different amount, something is wrong on the client and returning either answer is a guess. Refusing surfaces the bug instead of hiding it.

Idempotent-Replay: true on the response, so the client knows it is seeing a stored outcome and can re-fetch current state rather than trusting it as fresh — which is the staleness fix.

TTL of 7 days, chosen from the intent's lifetime, not from an expected retry delay. The question answered is "how long could a user reasonably still be trying to complete this one purchase".

And the product decision, written down where the next engineer will find it:

/**
 * IDENTITY: one key per user intent to pay, minted when the user confirms payment.
 * - Retries of that confirmation reuse it (network failure, app restart, background retry).
 * - A SECOND confirmation by the user is a NEW intent and MUST get a new key,
 *   even for an identical cart — buying the same coffee twice is legitimate.
 * TTL: 7 days. After that a retry becomes a new purchase.
 * Decided with payments + support, 2026-10-26. See ADR-031.
 */

How we proved it#

That a retry does not double-charge, and a repeat purchase is not blocked. The two cases the earlier fixes each got wrong, asserted together:

@Test
fun retryOfOneIntentChargesOnce() = runTest {
    val pending = PendingCharge(amountMinor = 2500, cartId = "c1")
    submit(pending); submit(pending); submit(pending)      // same key, three attempts
    assertEquals(1, gateway.charges.size)
}

@Test
fun aSecondUserConfirmationChargesAgain() = runTest {
    pay(coffeeCart)        // intent 1
    pay(coffeeCart)        // intent 2 — identical cart, new key
    assertEquals(2, gateway.charges.size)
}

key per network attempt:   test 1 FAILED (3 charges)   test 2 PASSED
key hashed from content:   test 1 PASSED               test 2 FAILED (1 charge)
key per intent:            test 1 PASSED               test 2 PASSED

That table is the post. Two tests, three definitions, and only one definition passes both.

That concurrent retries serialise:

@Test
fun concurrentRetriesReachTheGatewayOnce() = runTest {
    val pending = PendingCharge(amountMinor = 2500, cartId = "c1")
    (1..20).map { async { submit(pending) } }.awaitAll()
    assertEquals(1, gateway.charges.size)
}

read-then-write check:       FAILED — 3 charges
conditional reserve:         PASSED

And the production outcome:

double-charge support tickets / month
  before idempotency:     14
  key-per-attempt:        11      ← barely helped
  key-per-intent:          0

"could not buy twice" tickets
  content-hashed key:      23
  key-per-intent:           0

The middle row of the first table is the honest part: the original fix reduced double charges by about 20%, which looked like progress and was mostly noise, because the key definition was wrong.

One limit: the 7-day TTL is a judgement, not a measurement. We know retries almost never exceed a few hours; the 7 days is headroom for an offline client, and a genuine repeat purchase of the same cart within that window by the same user is distinguished only by being a new intent — which relies on the client minting correctly. A malicious or buggy client can still misuse it, which is what the 422 exists to surface.

What this generalizes to#

"Make it idempotent" is a request to define identity. The infrastructure is a hash map. The question — when are two things the same thing — is a domain question, and answering it with a technical default produces a technically correct system that is wrong about the business.

Place the identifier where the decision is made. The key belongs to the intent, not the transport. Same reasoning as a correlation id belonging to the user action rather than the HTTP request, or a transaction boundary belonging to the operation rather than the function.

Two indistinguishable requests mean you are missing a field. Retry versus repeat looked identical because nothing in the request carried intent. When a system cannot tell two cases apart, the answer is more information, not a cleverer heuristic.

Write the identity rule down next to the code. "What counts as the same request" is a decision with a rationale and stakeholders, and it is invisible in the implementation. Future engineers will otherwise re-derive it from the code and get it wrong, in one of the two directions above.

Related#

• The retry that turned a blip into an outage — the retries this protects against
• There is no UUID in the Kotlin standard library, and the fixes differ per platform — generating the key safely
• Your error message is an API, and changing it breaks someone — why the 422 carries a code

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.