A status column is wrong within a week

A hundred posts carried a field saying they were drafts. Sixty-eight of them were live, and had been for some time.

Nothing had gone wrong. Publishing happened in another system's interface, which writes nothing back, so the field recorded the last state anybody had typed.

The case this post is about#

One question — where is this post up to? — answered four ways by four different things:

on disk        the post file exists        → it has been written
frontmatter    status says ready           → the author considers it finished
ledger file    a local record of the push  → it reached the other platform
remote field   a url is present            → it is live

Only the second of those is a human judgement. The other three are observable facts, and before this split all four were a single hand-maintained state: field that somebody updated when they remembered.

So the field recorded what was true the last time anybody thought about it. A post could be live with the field saying ready, or deleted with the field saying live, and nothing anywhere disagreed.

This post follows those four rows, because three of them cannot drift and the fourth is the only one worth storing.

The symptom#

A figure that is quoted confidently and is wrong. "A hundred still to publish" was repeated in planning, used to decide what to work on, and nobody doubted it, because it came from a file in the repository rather than from somebody's memory — which felt like the same thing as coming from reality.

The second thing is that the error was not drift in the usual slow sense. The field was correct on the day each post was created and became wrong the moment a human published it elsewhere. There was no gradual decay to notice; the gap opened one post at a time and nothing anywhere compared the two sides.

The third is that the field looked maintained. Every post had one, the values were from a sensible vocabulary, and nothing was obviously stale or empty. A field that is uniformly populated with plausible values gives no signal at all about whether anybody has updated it recently.

The fourth is what made it durable. The only person positioned to update the field is the one doing the publishing, at the moment of publishing, in a different application — and the cost of switching back to a repository to edit a line of frontmatter is exactly high enough that it does not happen once the novelty wears off.

There is a general version of that worth stating: the person with the information is in the wrong application, and the cost of crossing between them is paid per item. Any convention that depends on somebody making that crossing a hundred times will hold for the first few and then quietly stop, and nothing in the record distinguishes the posts from before it stopped.

Why the obvious fix failed#

Remember to update it. The convention that produced the gap. It asks somebody who has just finished a task in one system to go and record that fact in another, which is the single most reliably skipped step in any workflow.

Update it in a script after publishing. Better, and it only covers publishes that go through the script. The whole reason this field was wrong is that publishing has a manual path — a person in a web interface — and a script cannot observe an action it was not part of.

Reconcile periodically. Useful and it answers the wrong question. A reconcile tells you the field was wrong; it does not stop it being wrong, and the interval between runs is a window in which every decision is made on stale data.

Make the field required in review. Pushes the burden to a moment when the information is not yet known. At review time the post has not been published, so the only honest value is "draft" — which is exactly what it said.

The actual mechanism#

Do not store the status. Compute it, from the artifacts, on every read.

on disk        the post file exists        → it has been written
frontmatter    status says ready           → the author considers it finished
ledger file    a local record of the push  → it reached the other platform
remote field   a url is present            → it is live

Each line is a fact about something that exists, checked at the moment somebody asks. None of them can be stale, because none of them is remembered — the state is a function of the files and the remote record rather than a value anybody typed.

The sub-case that makes this work in practice is having something to derive from on the remote side. A local file can tell you a post was pushed; only the remote can tell you it is live. That is why the push writes a small local record at the moment it happens, and why that record holds whatever identifier the remote returned: it is the hook that lets a later run ask the right question.

There is a distinction worth keeping that this collapses if you are careless. Two different things were being stored in one field: what the author intends — this draft is finished — and what the world is — this is published. The first is genuinely a stored value, because it is a judgement nobody can derive. The second is derivable and should never have been stored beside it.

Conflating them is what made the field feel defensible for so long. Half of it genuinely needed to be written down by a person, so the whole thing inherited that justification — and the derivable half rode along, inheriting an authority it had not earned. Splitting the two was most of the fix; once they were separate, nobody argued for storing the second.

The fix#

keep       exactly one stored field: the author's own judgement
derive     everything about publication from files and the remote
record     each push locally, at the moment it happens, with the returned id
report     the derived state on every run; never cache it into a column

One stored field, everything else computed from disk and the remote at read time:

// The ONLY thing a human sets. Everything else is observed.
type Stored = { status: "draft" | "ready" };

type Derived =
  | "planned"       // no file yet
  | "drafting"      // file exists, author has not called it ready
  | "ready"         // ready, nothing pushed
  | "remote-draft"  // pushed, not public
  | "remote-live";  // public, URL recorded

export function derive(p: Post, remote: Set<string>): Derived {
  if (!p.fileExists)                            return "planned";
  if (p.stored.status !== "ready")              return "drafting";
  if (p.meta.remote_url)                        return "remote-live";
  if (p.ledgerId && remote.has(p.ledgerId))     return "remote-draft";
  return "ready";
}

// Written at the moment of the push, from the RESPONSE — not from intent.
export async function recordPush(slug: string, res: { id: string; url: string }) {
  await Deno.writeTextFile(`.publish-out/${slug}.ledger.yaml`,
    toYaml({ slug, id: res.id, url: res.url, at: isoNow() }));
}

The drift a stored column produces, measured against the derived read:

$ status --stored          # what a status column claimed
  draft 100 · public 0

$ status                   # derived from disk + remote
  drafting 9 · ready 1 · remote-draft 75 · remote-live 168

Sixty-eight of those "drafts" had been published by hand, months earlier. Nothing updated the column because publishing happened in a web interface that had never heard of it.

The first line is what stops this becoming dogma. Not everything can be derived — intent cannot — and a system that tries to compute a judgement ends up inferring it from proxies. The rule is narrower than "never store state": it is that a fact about the world which can be observed should be observed rather than remembered.

What makes the derivation affordable is scale. A few hundred files and a few hundred remote records can be enumerated in under a second, so there is no performance argument for a cached column — and the cache is the thing that goes wrong.

The performance objection is worth meeting squarely anyway, because it is the one always raised. If the derivation ever does become expensive, the answer is to cache it with an explicit, visible expiry and a way to force recomputation — not to go back to a field somebody updates. Those are very different things that both get called caching.

How we proved it#

The discrepancy is countable and was counted. A hundred posts marked draft, sixty-eight of them live. That is the whole evidence, and it is the kind that settles an argument immediately: nobody defends a field after seeing the two numbers side by side.

It is worth running that comparison even when you are confident, precisely because confidence is the symptom. The field had been quoted in planning for weeks without challenge, and the only thing that produced the check was somebody needing the number for a different purpose and happening to look at the other side.

The derived version cannot reproduce the failure. There is no column to be stale, so the class of bug is gone rather than reduced. That is a stronger claim than "we fixed the data", and it is the only reason the fix is worth the work.

What I cannot tell you is when it started. A derived number has no history, and the stored field carried no timestamps, so the question "how long were we planning against a wrong figure" has no answer. That is a real cost of the approach and it is worth naming: a stored column is wrong and auditable, a derived one is right and has no past.

What I have not solved: anything that happens entirely outside the systems I can read. A post deleted in the remote interface, or renamed, leaves a local record pointing at something that no longer matches — and the reconcile direction catches that only if I run it. Derivation removes staleness within the boundary and does nothing beyond it.

The reconcile is therefore not optional so much as differently scoped. Deriving handles everything inside the systems you can observe; reconciling is how you learn about changes made outside them. Treating the two as alternatives — either derive or reconcile — gets it wrong in both directions.

And a limitation of removing the column: people liked the column. It was greppable, it appeared in diffs, and it could be read without running anything. Replacing it with a computation is better and it is also less convenient, and the honest answer to "can I just see the status" became "run this command" — which is a real usability regression that I accepted rather than solved.

The partial remedy was to have the command write its own output to a file in the repository, clearly labelled as generated and never read by anything that makes a decision. That restores the greppability without restoring the authority, and the distinction has to be held by convention because nothing stops somebody reading it.

The thing that reframed it was noticing what the field actually was. I had been treating it as data about the post. It was a record of somebody's belief at a past moment, which is a different kind of thing entirely — and once that was clear, the question stopped being "how do we keep this updated" and became "why is this stored at all".

What this generalizes to#

A stored fact about the world is a cache with no invalidation, and most of them are never described that way.

The pattern recurs anywhere a system records something it does not control. A user's subscription tier copied into a profile. An inventory count beside the rows it summarises. A deployment status written at deploy time. A "last seen" field updated by one of several paths. Each is a value that was true once, and each will diverge the moment the thing it describes changes without passing through the code that writes it.

So the question worth asking of any status field is what would have to happen for this to become wrong, and would anything notice — because if the answer to the second half is no, then the field is not describing the world, it is describing the last time somebody thought about it.

The test that makes this concrete is to ask who updates the field and in which application they are standing when they do it. If the honest answer involves switching tools, or involves somebody remembering after the fact, the field is already a record of intention rather than of state — and it will diverge at a rate set by how often that crossing is skipped.