The webhook that arrived twice, and the order that shipped twice

TL;DR#

Every webhook provider documents at-least-once delivery. Almost every handler is written as though it were exactly-once, because in testing it is.

Ours processed a payment.succeeded event twice — 40 seconds apart, same event id — and shipped the order twice. The duplicate was not a provider bug: our handler had taken 31 seconds to respond, the provider's timeout was 30, so it retried a request that had already succeeded.

Exactly-once is not something a provider can give you across a network. It is something a consumer constructs, with a deduplication key and a transaction, and the two have to be the same transaction.

The case this post is about#

A payment webhook handler, written the way they are:

@Post("/webhooks/payments")
suspend fun handle(@Body event: PaymentEvent): HttpResponse {
    when (event.type) {
        "payment.succeeded" -> {
            val order = orders.findByPaymentId(event.paymentId)
            orders.markPaid(order.id)
            fulfilment.ship(order.id)            // ← creates a shipment, charges postage
            email.sendConfirmation(order.id)
        }
        …
    }
    return HttpResponse.ok()
}

Correct for one delivery of one event. There is no duplicate handling because in every test there was exactly one delivery.

The symptom#

14:02:11  POST /webhooks/payments  evt_1K9x…  payment.succeeded
14:02:11  order 88214 → paid
14:02:14  shipment created: SHP-41822
14:02:42  handler returns 200          ← 31 seconds. the provider gave up at 30.
14:02:41  POST /webhooks/payments  evt_1K9x…  payment.succeeded     (retry #1)
14:02:41  order 88214 → paid           (idempotent by luck)
14:02:44  shipment created: SHP-41823  ← SECOND SHIPMENT
14:02:45  confirmation email #2 sent

Two shipments, two emails, one payment. The customer received two parcels.

Why the handler took 31 seconds: fulfilment.ship called a carrier API that was degraded that afternoon, taking 28 of the 31 seconds.

And the audit found it was not rare:

90 days of webhook deliveries
  total events received:                      412,880
  distinct event ids:                         408,114
  duplicate deliveries:                         4,766   (1.2%)
  duplicates that caused a double side effect:     38
    double shipments:    11
    double emails:       23
    double refunds:       4      ← the expensive ones

Thirty-eight real duplications, including four double refunds. The provider's dashboard had been reporting the retries all along under "delivery attempts".

Why the obvious fix failed#

"Respond faster." The right instinct, and it addresses the trigger rather than the class. We moved fulfilment to a background job, which cut response time to 80ms and reduced duplicates by about 90% — a provider can still deliver twice for its own reasons (a network partition, a deploy, an at-least-once internal queue), and 10% of 4,766 is still 476 duplicate deliveries.

"Check if the order is already paid." A guard we added:

if (order.isPaid) return HttpResponse.ok()      // already handled

It narrowed the window and did not close it. Two concurrent deliveries both read isPaid == false before either wrote — a read-then-write race, which is exactly what the 40-second retry was not, but which the faster handler then produced.

"Use a distributed lock on the event id." Prevents concurrency and not re-delivery. A retry an hour later acquires the lock cleanly and processes the event again.

"Ask the provider to guarantee exactly-once." They cannot. No provider can, across a network where an acknowledgement may be lost — and ours documents this clearly in the first paragraph of their webhooks page.

The actual mechanism#

The impossibility is the starting point: a sender cannot know whether a lost acknowledgement means the receiver processed the message.

sender → message → receiver   (receiver processes it)
sender ← ack ✗ lost
sender: "did they get it?" — unanswerable. So: retry.

Given that, a provider has two options, and every one of them picks the second:

At-most-once: send and never retry. Messages are lost. Unacceptable for a payment.

At-least-once: retry until acknowledged. Messages are duplicated. Acceptable, if the consumer deduplicates.

So exactly-once is not a delivery property; it is an effective processing property, and it is built at the consumer out of two things:

A deduplication key — a stable identifier for the logical event, which the provider supplies as an event id.

A transaction spanning the dedup record and the side effect. This is the part that is usually missing. Writing "processed evt_1K9x" and creating the shipment must succeed or fail together; if they are separate, a crash between them produces either a lost event or a duplicate one.

Exactly-once is a property of processing, not of delivery. It requires a dedup key and a transaction that contains both the dedup write and the effect — and if the effect is in another system, it requires an outbox.

The fix#

A dedup table written in the same transaction as the state change, with the external effects dispatched from an outbox.

CREATE TABLE processed_events (
    event_id    TEXT PRIMARY KEY,          -- the provider's id. PK = the dedup guarantee.
    event_type  TEXT NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    result      JSONB                       -- so a retry can return the ORIGINAL response
);

@Post("/webhooks/payments")
suspend fun handle(@Body event: PaymentEvent, req: HttpRequest): HttpResponse {
    verifySignature(req)                      // always first; see below

    return db.transaction {
        // 1. claim the event. a duplicate fails the PK constraint — atomically.
        val claimed = try {
            processedEvents.insert(event.id, event.type)
            true
        } catch (e: UniqueViolation) { false }

        if (!claimed) {
            val prior = processedEvents.get(event.id)
            return@transaction HttpResponse.ok(prior.result)   // replay the original response
        }

        // 2. the state change, in the SAME transaction as the claim
        val order = orders.findByPaymentId(event.paymentId)
        orders.markPaid(order.id)

        // 3. external effects go to an OUTBOX, not a direct call
        outbox.enqueue(ShipOrder(order.id))
        outbox.enqueue(SendConfirmation(order.id))

        val result = mapOf("orderId" to order.id)
        processedEvents.setResult(event.id, result)
        HttpResponse.ok(result)
    }
}

Five decisions:

The primary key is the dedup mechanism. An INSERT that violates it is an atomic test-and-set — no read-then-write race, no lock, no window. This is the whole fix in one line.

The claim and the state change share one transaction. A crash anywhere rolls back both, so the provider's retry finds the event unclaimed and processes it correctly. Separate transactions are the subtle bug that survives most first attempts.

External effects go through an outbox, because fulfilment.ship is in another system and cannot join the database transaction:

// a worker drains the outbox, at-least-once, and the DOWNSTREAM call is made idempotent
// with its own key — the same pattern, one layer down
suspend fun drain() = outbox.claimBatch(50).forEach { row ->
    when (val cmd = row.decode()) {
        is ShipOrder -> fulfilment.ship(cmd.orderId, idempotencyKey = "ship-${cmd.orderId}")
        is SendConfirmation -> email.send(cmd.orderId, idempotencyKey = "conf-${cmd.orderId}")
    }
    outbox.markDone(row.id)
}

The stored result is replayed on a duplicate. A retry gets the same response as the original, which is what makes the handler genuinely idempotent from the provider's point of view rather than merely harmless.

And signature verification happens before anything else, including dedup — an unverified event must not be able to insert a dedup row and thereby suppress the real one.

Plus a retention policy, since the table grows forever:

-- the provider retries for at most 72h; keep a week.
DELETE FROM processed_events WHERE processed_at < now() - interval '7 days';

How we proved it#

That a duplicate delivery is now a no-op, tested directly rather than hoped for:

@Test
fun aDuplicateDeliveryProducesNoSecondEffect() = runTest {
    val event = paymentSucceeded(id = "evt_test_1", paymentId = "pay_1")

    val first  = handler.handle(event)
    val second = handler.handle(event)        // same id

    assertEquals(HttpStatus.OK, first.status)
    assertEquals(HttpStatus.OK, second.status)
    assertEquals(first.body, second.body)      // replayed result
    assertEquals(1, outbox.count<ShipOrder>())
    assertEquals(1, outbox.count<SendConfirmation>())
}

That concurrent deliveries do not both win — the race the isPaid guard lost:

@Test
fun concurrentDuplicatesProduceOneEffect() = runTest {
    val event = paymentSucceeded(id = "evt_test_2", paymentId = "pay_2")
    (1..20).map { async { handler.handle(event) } }.awaitAll()
    assertEquals(1, outbox.count<ShipOrder>())
}

isPaid guard:        FAILED — 4 shipments (read-then-write race)
PK-based claim:      PASSED — 1

That a crash mid-handler does not lose or duplicate the event:

@Test
fun aCrashAfterTheClaimRollsBackTheClaim() = runTest {
    orders.failOnNextMarkPaid()
    assertFails { handler.handle(event) }
    assertNull(processedEvents.get(event.id))        // claim rolled back
    handler.handle(event)                             // the retry succeeds
    assertEquals(1, outbox.count<ShipOrder>())
}

That test is the one that justifies the single transaction, and it fails if the claim is committed separately.

And the production numbers over 90 days after the change:

                              before     after
duplicate deliveries           4,766      1,204   (handler now fast; provider still retries)
double side effects               38          0
double refunds                     4          0

Duplicate deliveries fell because the handler got fast; duplicate effects went to zero because of the dedup. The distinction matters: the first number is an optimisation and the second is the correctness fix.

The remediation of the original 38, because an honest writeup includes it:

double shipments:  11  → 9 recovered via carrier recall, 2 refunded (£340)
double emails:     23  → apology sent
double refunds:     4  → £1,180 recovered from 3 customers who were contacted; 1 written off

One limit: the outbox is still at-least-once to the downstream systems, so each of those needs its own idempotency key — which is why fulfilment.ship takes one. If a downstream provider has no idempotency support, exactly-once stops at your database boundary and the effect can still double. Ours both support it; one legacy internal service does not, and for that one we dedupe on a natural key in the receiving service instead.

What this generalizes to#

At-least-once plus consumer deduplication is how exactly-once is built. This is true for webhooks, message queues, job runners, retried RPCs and mobile sync. Any time delivery can be retried, the consumer owns the dedup — and a provider promising exactly-once is either deduplicating for you or mistaken.

A unique constraint is an atomic test-and-set. INSERT and catch the violation is simpler, faster and more correct than read-then-write, and it needs no lock. This is the single most underused concurrency primitive in application code.

The dedup record and the effect must share a transaction. Two transactions means a crash between them loses or duplicates. When the effect is in another system, an outbox is how you restore the single transaction — and the outbox then needs its own idempotency downstream.

Replay the original response, do not just return OK. A caller retrying because it lost your answer needs the answer. Storing the result turns "harmless duplicate" into "genuinely idempotent", which is what the retrying system was asking for.

Related#

• Idempotency is a product decision before it is a technical one — choosing the dedup key
• The retry that turned a blip into an outage — why the provider retried at all
• Every timeout you set independently adds up to one that is wrong — the 30-second timeout that triggered it

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.