Five ways a cache and its writes drift apart
A cache and the writes against it drift apart in five distinct ways. Each ships separately, each looks reasonable, and all five look identical to a user.
A cache in front of a remote source is one of the most common structures in an application, and the interesting failures are not in the cache. They are in the relationship between the cache and the writes that go past, around or through it.
There are five distinct ways that relationship breaks. I have met all five in the same codebase.
The case this post is about
One profile screen showing a name the user changed an hour ago, and the five independent reasons it could be doing that:
bypass the write did not go through
read split two routes, only one of which goes through
ordering writes applied in an order nobody chose
bookkeeping the record of pending writes is not connected
cold start age is not part of what is served
Every one of those five shipped in the same codebase. Each was added by somebody solving a real problem, each is defensible on its own, and all five produce the identical user-visible symptom — the app showing something that is no longer true.
So the bug report is always the same sentence and the cause is one of five places. This post follows that one screen, because the five are not variations on a theme; they are five distinct defects that happen to be indistinguishable from the outside.
The symptom
To a user, all five are the same event: the app displays something that is not true. A list missing the item they just created, a value that reverts after a moment, a detail screen disagreeing with the list it was opened from, a toggle that flips back.
The second thing is that each one is individually small. None of these is an architectural failure; each is a plausible decision taken under time pressure, usually by somebody who knew the system well. They accumulate because nothing connects them — there is no review in which five separate commits across eight months appear together.
The third is that they are hard to reproduce. Every one of them is a timing or ordering property, so the reproduction involves doing two things in a particular sequence or at a particular moment, and the bug report says "sometimes the list is wrong".
The fourth is that the cache looks innocent throughout. Inspect it and it contains what it was told to contain. The defect is always in what it was told, or when, or by whom — which means reading the cache implementation is the wrong place to spend the first hour, and it is where everybody spends it.
The five
Mutations that bypass the cache. A write goes straight to the remote source and succeeds. The cache is not told, so it keeps serving the previous value until something else invalidates it. This is the most common one and the most defensible in review — the write is correct, it just has a second responsibility nobody mentioned.
Inconsistent read paths. Two screens ask for the same data by different routes: one through the cache, one directly. They disagree whenever the cache is stale, which is sometimes, and the disagreement is between two screens rather than between a screen and the truth — so it reads as a UI bug.
Unordered retries. A write fails and is retried. Meanwhile another write to the same record succeeds. The retry lands afterwards and reinstates the older value, correctly executing a request that is no longer wanted. Nothing here is a bug in isolation; the queue simply has no opinion about ordering.
A write bookkeeper that is not wired in. The mechanism for tracking pending local writes exists, is correct, and is not connected to the path that actually performs them. Every piece is present and the link is missing, so the system behaves as though it has no bookkeeping — while containing the code that would do it.
Cold start serving stale data as fresh. On launch, the cache has yesterday's content and no indication that it is old. The first screen renders it immediately, which is the desired behaviour, and nothing distinguishes "this is cached and being refreshed" from "this is current". The user acts on it.
Why the obvious fix failed
Clear the cache more aggressively. The reflex, and it converts a correctness problem into a performance problem while only partially solving the correctness problem. Clearing on every write makes every screen slow and still leaves the ordering and cold-start cases untouched.
Add a refresh button. Moves the responsibility to the user, who has no way to know whether what they are looking at needs refreshing. It is also an admission in the interface that the application does not know its own state.
Reach for a different library. Tempting when five problems appear in one area, and wrong, because four of the five are not properties of the caching mechanism at all. They are properties of how writes are routed, ordered and recorded around it. A new library arrives with the same five opportunities.
Fix them one at a time as they are reported. What actually happens, and it is why they coexist. Each is reported separately, fixed separately, and the fix is scoped to the symptom — so the next one arrives looking unrelated.
The actual mechanism
The five have one shape: something other than the cache decided what the truth was, and the cache was not part of that conversation.
bypass the write did not go through
read split two routes, only one of which goes through
ordering writes applied in an order nobody chose
bookkeeping the record of pending writes is not connected
cold start age is not part of what is served
Four of the five are fixed by the same structural rule: every write goes through the same path, and that path is responsible for both the remote call and the local state. Not "remember to update the cache" — a single entry point through which a write cannot avoid doing both.
That rule fixes bypass by construction, fixes the read split by giving reads one route to mirror, and gives ordering and bookkeeping a place to live — because a single write path can hold a queue and a record of what is pending, where five scattered call sites cannot.
The fifth is different and needs its own answer: age has to be part of what is served. A cached value delivered without any indication of its age cannot be distinguished from a current one by anything downstream, including the person looking at the screen. The read path has to be able to say this is cached, a refresh is in flight — which is a different statement from this is the data.
The sub-case that is easiest to underestimate is ordering. It is tempting to treat a retry as idempotent — same request, same result — and that is true of the request and false of the sequence. A retry is a request from the past arriving in the present, and whether it should still be applied depends on what happened in between, which is a question the retry mechanism has no way to ask unless the write path keeps enough history for it to be answerable.
The fix
one write path remote call and local state, inseparably
reads mirror it every screen reads through the same route
ordered queue writes applied in a known order, retries aware of what followed
bookkeeper wired the pending-write record connected to the path that performs them
age in the read cached-and-refreshing is a distinct state from current
All five in one path. The write goes through the store, the bookkeeper is wired to the thing that performs the writes, and the read carries its own age:
class ProfileStore(
private val api: ProfileApi,
private val dao: ProfileDao,
private val bookkeeper: Bookkeeper<String>, // WIRED, not merely constructed
) {
// ONE write path. A caller cannot reach `api` or `dao` directly.
suspend fun write(p: Profile) {
dao.upsert(p.copy(pending = true)) // local first, visibly pending
bookkeeper.record(p.id) // queued, in order
runCatching { api.put(p) }
.onSuccess { dao.upsert(p.copy(pending = false)); bookkeeper.clear(p.id) }
.onFailure { /* stays pending; the queue retries in order */ }
}
// Reads MIRROR the write path, and carry age rather than hiding it.
fun stream(id: String): Flow<Aged<Profile>> = flow {
dao.get(id)?.let { emit(Aged.Cached(it, pending = it.pending)) }
runCatching { api.get(id) }
.onSuccess { dao.upsert(it); emit(Aged.Fresh(it)) }
.onFailure { e -> dao.get(id)?.let { emit(Aged.Stale(it, e)) } }
}
}
The anti-pattern this replaces, and what each costs:
mutation bypasses store write hits api directly -> cache never updated
inconsistent reads one screen reads dao, another reads api -> they disagree
pagination race page 2 arrives before page 1 -> list order is arrival order
bookkeeper unwired constructed, never consulted -> pending writes never retried
cold-start stale Cached indistinguishable from Fresh -> yesterday shown as current
The fourth is the one that passes every test. A bookkeeper that is injected, constructed and never consulted satisfies DI, satisfies a unit test that checks it records, and silently drops every offline write.
The thing that makes this tractable is that it is mostly one decision rather than five fixes. Four of
the problems disappear when writes have a single path; the fifth needs the read side to carry age.
Taken as five separate bugs it is eight months of work, and taken as one structural property it is a
refactor with a clear shape.
The order you fix them in is worth thinking about, because it is not the order they were reported in.
The single write path has to come first — not because it is the most urgent, but because the other
work has nowhere to live until it exists. A queue needs a path to sit in; a bookkeeper needs
something to record. Attempting the ordering fix before the structural one means building a queue in
front of several call sites, which is most of the work and none of the benefit.
That is the practical reason these linger. Taken individually, each has a cheap local fix and an
expensive structural one, and the cheap fix always wins on the day it is reported. The structural work
only looks proportionate once somebody has listed all five together, which is the step nothing
prompts.
## How we proved it
**All five were present in one codebase**, which is the finding that makes this worth writing down.
Not as a survey of what can go wrong in theory — as five things that had each independently seemed
reasonable, shipped, and coexisted. That co-occurrence is the evidence that they share a cause: they
are not five unrelated bugs, they are five expressions of the cache not being in the write path.
**The structural claim is checkable rather than argued.** Whether every write goes through one path
is a question about the code that can be answered by looking: count the places that call the remote
source directly. If the answer is more than one, bypass is possible regardless of anybody's intent.
**What I cannot give you is a rate.** I do not know how often each of these produced a visible wrong
value for a user, because the symptom is indistinguishable from the others and the reports said "the
list was wrong". The five are distinguishable in the code and were not distinguishable in the bug
tracker, which is itself part of why they survived.
**What I have not solved:** the ordering case properly. A write path that holds a queue makes
ordering *possible* to reason about; deciding what a retry should do when the world moved underneath
it is a policy question I have answered case by case rather than generally. There is a real design
there — and what I have is a queue and some judgement.
**And a limitation of the single-write-path rule:** it is enforceable by convention and by review, and
I have not made it mechanical. Nothing fails if somebody adds a sixth call site to the remote source
tomorrow. The check is straightforward to describe — no direct calls outside the write path — and
until it exists, the structure holds because people remember it, which is the kind of guarantee this
whole post is complaining about.
## What this generalizes to
When one area produces several unrelated-looking bugs over time, the useful question is not how to
fix each one. It is **what single property they would all be consequences of.**
The tell is that they are reported separately, fixed separately, and keep arriving. That pattern says
the fixes are landing at symptom level, because a cause-level fix would have closed several at once.
Five bugs in eight months in one subsystem is information about structure, and it is information that
only becomes visible if somebody lists them together — which nothing in a normal workflow prompts
anyone to do.
So when the third one arrives, write the first two down beside it. The grouping is the diagnostic
step, and it costs nothing; the alternative is fixing the fourth and fifth individually as well, each
time with a solution that is locally correct and does not generalise.
One post a week, no digest
New writing when it lands. No roundups, no course pitch, no "10 tips".