A retry queue with nothing to run it is a graveyard

A write fails because the device is offline. The record is kept locally and marked as needing sync, which is exactly right, and the user sees their change.

Nothing ever retries it. The mark is a note to a process that was never written.

The case this post is about#

One pending-writes queue that only ever grew.

Marking a failed write for retry was already there and worked. What was missing is everything that makes a retry happen:

signal       something that reports connectivity changes, not just current state
enumerate    a query that lists what failed, ordered, with attempt counts
backoff      increasing delays per record, with a ceiling and a declared outcome

Without the first, retries ran when the app happened to open. Without the second, the queue was drained in whatever order the database returned. Without the third, a record that could never succeed was retried forever and blocked everything behind it.

This post follows that queue, because each of those three is individually obvious and the failure mode of omitting any one of them is the same: writes that are recorded, marked, and never sent.

The symptom#

Data that never reaches the server. Not a visible failure — the local write succeeded, the screen updated, and the record carries a flag saying it still needs sending. Weeks later it is still carrying it, along with several hundred others.

The second thing is that it looks fine from inside the app. Every read comes from local storage, so the user's own device shows all their data, correctly, indefinitely. The absence only becomes visible on a second device, after a reinstall, or when somebody looks at the server — which is to say, at the worst possible moment.

It is also the reason this survives testing. Every manual check is performed on the device that made the change, where the data is present and correct by construction — so the bug is invisible to exactly the procedure most likely to be run against it.

The third is what usually passes for a retry. There is often something: a sync on app launch, or a pull-to-refresh that happens to push too. Those fire on user actions rather than on circumstances, so a device that regains connectivity while the app is backgrounded does nothing, and the retry is coupled to somebody opening a screen.

The fourth is that the queue has no shape. Failed records accumulate with no ordering, no attempt count and no age; when something does eventually run, it tries all of them, hits the same failure for whatever is genuinely unsendable, and either stops at the first error or hammers the server with several hundred requests at once.

Why the obvious fix failed#

Retry on app launch. Better than nothing and it ties data integrity to a user action. The records that need sending most urgently are on a device whose owner is not currently using the app, which is precisely the condition under which this never fires.

Retry immediately on failure, with a short delay. The instinct, and it is wrong about why the write failed. A retry three seconds after an offline failure fails the same way, and the loop either gives up early or burns battery discovering the same thing repeatedly.

Retry everything whenever connectivity returns. The right trigger and a dangerous payload. A device that has been offline for a week reconnects and issues several hundred requests simultaneously — which is a self-inflicted load spike, and if the server rejects some of them under that load, the failures are indistinguishable from the original ones.

Add an attempt counter and give up after N. Necessary and not sufficient on its own, because giving up silently converts an unsent record into a lost one. The counter is right; what is missing is what happens at the ceiling, which is a product decision rather than a technical one.

The actual mechanism#

A retry queue needs three separate parts, and the common failure is building one of them and calling it a mechanism.

signal       something that reports connectivity changes, not just current state
enumerate    a query that lists what failed, ordered, with attempt counts
backoff      increasing delays per record, with a ceiling and a declared outcome

The first is the part most often missing. A flag on a record is a statement about the past; a connectivity signal is what turns that statement into an action at the moment it can succeed. Those are different responsibilities and neither implies the other, which is why a codebase can have careful failure marking and no retry behaviour at all.

The second line matters because of what it prevents. An enumeration that returns records in a defined order, with their attempt counts, lets the runner process them deliberately — oldest first, a bounded number at a time — rather than discovering the whole backlog at once. Without it, every run is all-or-nothing.

The sub-case that has to be decided rather than defaulted is the ceiling. A record that has failed twenty times is not going to succeed on the twenty-first, and the choices are to surface it to the user, to discard it with a record of having done so, or to park it in a state somebody can inspect. All three are defensible. Doing none of them — retrying forever — is the one that is not, and it is the default.

There is a related decision about ordering that matters more than it looks. Retrying the oldest first is the obvious choice and it means a single permanently-failing record at the head of the queue blocks everything behind it. Either the runner skips past a record that has exceeded its attempts, or the backlog has one poison entry and stops.

The poison entry is worth expecting rather than discovering, because it is close to guaranteed over a long enough backlog. Any set of several hundred deferred writes will contain at least one that refers to something deleted, or was authored under credentials since revoked, or was malformed by a version of the app nobody runs any more.

The fix#

observe     connectivity transitions, and run on the transition
enumerate   failed records in a defined order, with attempt counts
back off    per record, increasing, with a ceiling
decide      what the ceiling means — surface, discard with a record, or park
skip        a record over its limit, so it cannot block the queue behind it

The per-record attempt count is what makes the last line possible — a queue with one shared retry counter cannot tell a poisoned record from a flaky network:

class SyncOrchestrator(
    private val queue: PendingDao,
    private val api: SyncApi,
    private val clock: Clock,
) {
    suspend fun drain() {
        for (rec in queue.pending().sortedBy { it.queuedAt }) {    // defined ORDER
            // Per-record backoff: 1s, 2s, 4s … capped. A record not yet due is
            // skipped, NOT waited on — waiting blocks everything behind it.
            val due = rec.lastAttempt + minOf(1L shl rec.attempts, 300L) * 1000
            if (clock.now() < due) continue

            if (rec.attempts >= MAX_ATTEMPTS) {
                queue.park(rec.id, reason = "exceeded $MAX_ATTEMPTS attempts")
                continue                                   // parked, not silently dropped
            }

            runCatching { api.push(rec.payload) }
                .onSuccess { queue.clear(rec.id) }
                .onFailure { queue.recordAttempt(rec.id, at = clock.now()) }
        }
    }
    private companion object { const val MAX_ATTEMPTS = 8 }
}

A queue with one record the server will never accept:

$ drain
  rec-7   attempt 8/8 -> PARKED (exceeded 8 attempts)
  rec-8   ok
  rec-9   ok
  3 processed · 1 parked · 0 blocked

Without the park, rec-7 retries at the head of the queue forever and rec-8 and rec-9 never sync. Without the per-record counter, one genuinely-poisoned record exhausts the shared budget and everything else is abandoned with it.

The fourth line is the one that turns this from a technical mechanism into a complete one. Every
other line is a matter of making the retry work; that line is about what you owe the user when it
cannot, and it is the part that gets deferred because it requires a decision somebody has to make
rather than code somebody can write.

What keeps the implementation small is that none of the three parts is novel. The platform supplies
a connectivity signal, the local store already supports a filtered query, and backoff is
arithmetic. The work is in wiring them together and in the one product decision — which is why the
usual outcome is one part built, working, and inert.

That shape is worth recognising on its own, because it is not a quality problem. The part that
exists is usually well written; it is simply not connected to anything that causes it to execute.
Reviewing it finds nothing wrong, because nothing is wrong with it.

## How we proved it

**The accumulation is directly countable.** Query the local store for records still marked as
needing sync and look at the oldest timestamp. If there are hundreds and the oldest is weeks old,
nothing is running — and that number is the entire diagnosis, available in one query on any device
with the problem.

**The connectivity-transition requirement is demonstrable.** Put the device in airplane mode, make a
change, restore connectivity without touching the app, and wait. If the record is still unsent, the
retry is coupled to a user action rather than to circumstance, which is the condition this is
about.

![All at once and still waiting, then drained in bounded groups](/posts/sync-orchestrator-with-backoff/anim-drain.gif)

**What I cannot tell you is how much data was lost**, as opposed to delayed. Records that were still
queued when I looked were recoverable; records on devices that were reinstalled before anybody
noticed are not countable, by definition. The honest framing is that the mechanism was broken and
the loss is unquantified rather than zero.

**What I have not solved:** conflict on eventual send. A record that failed to sync for a week may
arrive after the same entity was modified elsewhere, which is a merge question this mechanism does
not answer — it gets the write delivered and says nothing about whether it should still win. That
is a separate decision and it needs to exist before a long backlog drains.

The ordering between the two matters practically. A retry mechanism shipped without a conflict
policy will, the first time it drains a week-old queue, overwrite newer data with older and do it
confidently — so the retry is the more dangerous half to ship first, which is the opposite of how
it usually gets prioritised.

**And a limitation of backoff specifically:** it optimises for a server that is temporarily
unavailable, and a large share of real failures are not that. A malformed record, a revoked token
or a deleted parent will fail identically on every attempt, and exponential backoff just means
discovering that more slowly. Distinguishing retryable failures from permanent ones is the
refinement I would build next, and I have not.

The crude version of it would get most of the benefit: treat a transport-level failure as retryable
and a rejection by the server as permanent, on the grounds that the server has already looked at the
payload and declined it. That is wrong in some cases and it is a far better default than treating
every failure as temporary, which is what backoff alone implies.

**The thing I would tell somebody** is to check which of the three parts exists before concluding
there is a sync problem. In every instance I have looked at, the marking was careful and correct
and one of the other two was simply absent — and that is a much smaller thing to fix than the
symptom suggests.

The diagnosis order that works is the reverse of the instinct. Rather than starting at the failing
write and tracing forward, start at the question "what process would run this, and what wakes it" —
and if either answer is missing, stop, because everything upstream of it is already working.

## What this generalizes to

**Recording that something needs doing is not the same as arranging for it to be done**, and the
first is so much easier that it routinely stands in for the second.

The pattern is everywhere. A dead-letter queue nothing consumes. A table of pending jobs with no
worker. A cleanup flag no sweep reads. A scheduled task registered against a scheduler that was
never started. In each case the bookkeeping is immaculate and the work does not happen, and the
bookkeeping is reassuring enough that nobody checks.

So the question worth asking of any deferred work is **what runs it, and what causes that to run** —
because a queue is a data structure, a runner is a process, and a trigger is a decision about
circumstance. Having one of the three is the normal condition, and it looks from the inside exactly
like having all of them.