The architecture decision nobody wrote down, re-litigated four times

TL;DR#

We decided to keep the mobile client's sync queue in SQLite rather than in memory. The decision was right, it was made carefully over two days, and it was never written down.

It was then re-opened four times in two years, by four different engineers, each of whom independently proposed the in-memory design and each of whom had to be talked out of it by whoever still remembered. Total cost: roughly eleven engineer-days of discussion and one half-built prototype.

What makes a decision stay decided is not the decision. It is the rejected options and why they were rejected — because that is what the next person needs and the code cannot contain.

The case this post is about#

The original decision, in a two-day discussion across a thread and a whiteboard:

Sync queue: persist pending operations in SQLite rather than holding them in memory.

The implementation is unremarkable:

class SyncQueue(private val db: Database) {
    suspend fun enqueue(op: PendingOp) = db.pendingOpQueries.insert(op.toRow())
    suspend fun next(): PendingOp? = db.pendingOpQueries.oldest().executeAsOneOrNull()?.toOp()
    suspend fun complete(id: Long) = db.pendingOpQueries.delete(id)
}

What the code does not contain: that an in-memory queue was tried first, and that it was abandoned because operations were lost on process death — which on Android is routine, not exceptional, and which the product could not tolerate for offline edits.

The symptom#

The same proposal, four times:

year 1, Q3   engineer A: "why is the queue in SQLite? a List<PendingOp> would be simpler
             and we'd avoid the serialisation" — 2 days of discussion
year 1, Q4   engineer B: proposed it in a design review for the new editor — 1 day
year 2, Q1   engineer C: BUILT IT. A branch replacing the queue with an in-memory
             implementation, 4 days of work, closed after review
year 2, Q3   engineer D: raised it again, having read engineer C's closed PR and
             concluding it was closed for "style reasons" — 2 days

Eleven engineer-days and one four-day prototype, all re-deriving a conclusion reached in year zero.

Three things made each round expensive:

The proposal was good. An in-memory queue is simpler, and the serialisation cost is real. Every proposer was reasoning correctly from the information available.

The counter-argument lived in one person's memory. Specifically mine, and when I was on leave during year 2 Q1, engineer C built the branch unopposed.

And the closed PR looked arbitrary. My review comment said "this loses operations on process death — we tried this". Engineer D read that two quarters later and could not tell whether it was a measured finding or an assertion, so they reasonably re-opened it.

Why the obvious fix failed#

"Explain it in a code comment." We added one:

// Persisted in SQLite: an in-memory queue loses operations on process death.

Engineer D had read it. A single sentence asserts a conclusion without the evidence, so it does not settle anything — and the question it provokes is "how often does that actually happen?", which the comment cannot answer.

"Point people at the original thread." The thread was in a chat tool with a 90-day retention policy. It no longer existed, which is a detail worth noticing: decisions recorded in chat have an expiry date.

"Add it to the team wiki." There was a wiki page, three levels deep, last edited eighteen months earlier. Nobody found it because nobody searches a wiki for "why is this like this" — they ask the code, then a person.

"Tell new joiners during onboarding." We did, and this is one decision among dozens. Verbal transmission degrades, and engineer C joined after the person who did the onboarding had left.

The actual mechanism#

Code records what was decided. It cannot record three things the next person needs:

The options rejected. The in-memory design is absent from the codebase. There is no trace that it was considered, so it looks like an unexplored alternative rather than a tested one.

The reason, with evidence. "Loses operations on process death" is a claim. How often? Under what conditions? Measured how? Without that, it reads as a guess, and a reader with a good idea will weigh their own reasoning above an unsupported assertion — correctly.

The constraints at the time. Which may have changed. If they have, re-opening the decision is right, and you cannot tell without knowing what they were.

So the default is re-litigation, and it is not a failure of discipline — it is the rational response to a conclusion with no visible justification:

reader sees:  a design + a one-line assertion
reader has:   a coherent argument for a simpler design
rational move: raise it

Which makes the quality bar for a decision record checkable rather than a matter of taste. A record that states only its conclusion is the thing that invites re-litigation, so that is exactly what the check looks for:

import re

REQUIRED = ["Context", "Options considered", "Decision", "Consequences"]

def review(adr_text: str):
    problems = []
    sections = {m["name"]: m["body"] for m in re.finditer(
        r"^## (?P<name>.+?)\n(?P<body>.*?)(?=^## |\Z)", adr_text, re.M | re.S)}

    for name in REQUIRED:
        if name not in sections:
            problems.append(f"missing section: {name}")

    options = sections.get("Options considered", "")
    # One option is not a comparison — it is a conclusion with a heading above it.
    # This is the single most common shape, and the one that re-litigates.
    rejected = re.findall(r"^### (?!.*\bChosen\b).+", options, re.M)
    if len(rejected) < 1:
        problems.append("no REJECTED option documented — nothing records what was ruled out")

    # A reason without a measurement is an assertion, and a reader with a good argument
    # will correctly outweigh it. Look for a number, a unit, or a named observation.
    if not re.search(r"\d+\s*(ms|s|%|rps|MB|GB|/day|per day)|measured|benchmark", adr_text, re.I):
        problems.append("no evidence — the rationale is assertion, not measurement")

    # Constraints expire. A record with no date cannot tell a reader whether re-opening
    # the decision is now CORRECT, which is a legitimate reason to re-open it.
    if not re.search(r"^Date:\s*\d{4}-\d{2}-\d{2}", adr_text, re.M):
        problems.append("no date — a reader cannot tell which constraints still hold")

    return problems

Run against the record that was eventually written for the queue decision, first draft:

$ adr-review docs/adr/0014-durable-job-queue.md

no REJECTED option documented — nothing records what was ruled out
no evidence — the rationale is assertion, not measurement

$ adr-review docs/adr/0014-durable-job-queue.md     # after adding the in-memory option
                                                     # and the crash-rate measurement
ok

The first run is the interesting one, because the draft felt complete — it had context, a decision and consequences, and read like a proper document. What it lacked were the two things that make a conclusion survive contact with a smart reader: the alternative that was tested, and the number that settled it.

A decision stays decided when the rejected options and the evidence are written down. The conclusion alone invites a well-reasoned challenge, because a reader cannot distinguish a measured finding from an opinion.

The fix#

One ADR, written in forty minutes, two years late.

# ADR-017: Persist the sync queue in SQLite

Status: accepted · Date: 2024-02-11 (written 2026-03-04, see Note)
Deciders: @rajan, @priya, @tom

## Context
Pending offline operations must survive the app process being killed. On Android,
process death while backgrounded is routine: measured on our own fleet, 31% of
sessions end in process death rather than a user-initiated exit (telemetry,
Jan 2024, n=84,000 sessions).

## Options considered

### A. In-memory queue (List<PendingOp>)  — REJECTED
Simplest, no serialisation cost (~0.4ms/op measured).
Rejected because: operations are lost on process death. We shipped this in
v1.2 and measured 1,840 lost edits over 9 days (0.7% of all offline edits),
producing 212 support tickets. See incident 2024-01-29.

### B. SQLite-backed queue  — ACCEPTED
Survives process death. Costs ~0.4ms per enqueue and a schema to maintain.
Chosen because durability is a product requirement for offline edits, and
0.4ms is not observable to a user.

### C. SharedPreferences / DataStore  — REJECTED
Durable, and not transactional across the queue and the entity it mutates,
so a crash between the two leaves an orphaned operation. SQLite gives us
both in one transaction.

## Consequences
- Every PendingOp needs a serialisable form and a schema migration path.
- The queue can be inspected in a debug build, which has been useful twice.
- If the platform ever guarantees process survival (it will not), revisit.

## Revisit if
- Offline editing stops being a product requirement.
- Measured process-death rate falls below ~2% of sessions.

## Note
Written retroactively after this decision was re-opened four times
(~11 engineer-days). The numbers above are from the original 2024 telemetry.

Four things make it work, and they are the parts people leave out:

Option A has the measurement. 1,840 lost edits, 0.7%, 212 tickets. That is what makes the rejection a finding rather than an assertion, and it is what engineer D needed.

Option C exists. Nobody had considered DataStore in the original discussion; writing the ADR surfaced it and the transactionality argument. Documenting a decision sometimes improves it.

"Revisit if" names the conditions. The decision is not permanent, and the ADR says what would make re-opening it correct. That converts "stop asking" into "here is when to ask".

And the retroactive note is honest about its own lateness, with the eleven days recorded, because that cost is the argument for writing the next one on time.

Then the process change:

## When to write an ADR
Write one when a decision (a) is hard to reverse, (b) had a plausible
alternative, or (c) will look arbitrary to someone who was not in the room.

Where: docs/adr/NNN-slug.md, in the repo, next to the code it governs.
Length: one page. Options and evidence are mandatory; prose is not.
Link it from the code:
    // See docs/adr/017-sqlite-sync-queue.md for why this is persisted.

The code comment now points at the evidence instead of asserting the conclusion.

How we proved it#

That it stopped the re-litigation — the only outcome that matters, measured over the following eighteen months:

proposals to move the queue in-memory
  two years before the ADR:   4   (11 engineer-days + a 4-day prototype)
  eighteen months after:      1

And that one is the interesting case:

year 3, Q2   engineer E read ADR-017, checked the "revisit if" condition against
             current telemetry (process-death rate now 26%, still well above 2%),
             and closed their own question in 20 minutes without raising it.

Twenty minutes instead of two days, and self-served. That is what the "revisit if" section is for.

Adoption across the team:

ADRs written in the 18 months after:            23
decisions re-opened that had an ADR:             1
decisions re-opened that did not:                9

Nine of ten re-litigations were on undocumented decisions. Small numbers, and the direction is stark.

The cost of writing them:

median time to write an ADR:        35 min
median length:                      1.1 pages
ADRs later superseded (correctly):  4 of 23   ← the process working

Four were superseded, which is the right outcome — an ADR is not a commitment, and a decision revisited with its original evidence is a much better conversation than one revisited without.

One honest limit: ADR-017's numbers came from 2024 telemetry that I had to reconstruct from an old incident report, and one figure (the 0.4ms serialisation cost) I re-measured in 2026 rather than recovering the original. Writing an ADR two years late means some of its evidence is archaeology — which is an argument for writing them when the decision is made, not an argument against writing them late.

What this generalizes to#

Record the rejected options, not just the choice. The alternatives are invisible in the artifact and are exactly what the next person will propose. A decision document without them does not prevent anything.

Evidence is what makes a conclusion durable. "We tried it and lost 1,840 edits" settles a question that "an in-memory queue loses operations" cannot. Numbers are what distinguish a finding from an opinion in the reader's eyes, and the reader is right to make that distinction.

Name the conditions for revisiting. Decisions are made under constraints, and constraints change. A "revisit if" section turns a closed question into a monitorable one, and it is what lets someone self-serve instead of calling a meeting.

Put it next to the code and link to it. A wiki three levels deep is not found; docs/adr/ in the repository, referenced from a comment at the relevant line, is found by the person who has the question at the moment they have it.

Related#

• The interface with one implementation, and the three years it cost — a decision that should have been an ADR note
• The code review that should have been a conversation — where the decision gets written back
• A stale doc does more damage than a missing one — keeping records honest

Keep reading#

This is part of Architecture decisions & code review — how choices get made, recorded, and revisited. Follow MobileByteSensei for the rest.