The numbers worth watching, and the ones that flatter

Every piece of advice about blog metrics says the same thing: views flatter, engagement is better, and the numbers worth tracking are the durable ones.

That is true and it is the easy half. The harder problem is a number that is measuring the right thing and computing it wrong.

The case this post is about#

One status board, and the two different ways its numbers could lie:

type State = "planned" | "drafting" | "ready" | "remote-draft" | "remote-live";
/** Disk and remote are the sources. There is no stored status column to drift. */
export function stateOf(p: Post, remoteIds: Set<string>): State {
  if (!p.fileExists)                            return "planned";
  if (p.meta.status !== "ready")                return "drafting";
  if (p.meta.remote_url)                        return "remote-live";

The first kind is familiar: a stored column that nobody updated, so the board says ready about a post somebody has since rewritten. Deriving from disk removes it.

The second kind survives the fix and is harder to see. remote-live is derived from a rule — presence of a URL — and the rule is wrong for a post that was published and then unpublished. The number is computed fresh every run and is still false.

This post follows that function, because the second kind of wrong number looks exactly like a correct one.

The symptom#

A status column said a hundred posts were drafts. Around sixty of them were live and had been for some time. The field was not stale in the ordinary sense — nothing had updated it because nothing could: publishing happened in another system's interface, which writes nothing back, so the local record described the last state anybody had typed.

The second instance was a backlog count. A tool reported fifty-six posts still to write, derived by a rule that looked sound — a planned row with no file on disk is unwritten. Eleven of those fifty-six had been written under a different filename, so the rule found no file and reported them as outstanding. The number was not an estimate that drifted; it was a computation that was wrong by twenty per cent, reported to the character.

The third thing both have in common is that nothing contradicted them. A wrong count does not produce an error, it produces a count — confident, specific, and indistinguishable from a right one. The only reason either was found is that somebody went looking for an unrelated reason.

The fourth is that these are worse than vanity metrics, and the difference is worth being precise about. A view count is a real measurement of a thing that does not matter much; everybody knows to discount it. A derived count is a measurement of a thing that matters a great deal, presented with the authority of arithmetic, and nobody discounts it at all.

The asymmetry shows up in how each gets challenged. Somebody quoting a view count in a meeting will be asked what it means; somebody quoting "fifty-six remaining" will be asked what order to do them in. The second number has skipped the stage where anybody interrogates it, which is exactly the stage that would have caught it.

Why the obvious fix failed#

Track better metrics. The standard advice, and it addresses the first failure mode only. Swapping views for indexed pages is an improvement and does nothing about whether the indexed-page number is computed correctly — it moves the attention to a different figure with the same class of risk.

Store the status and keep it updated. What produced the hundred-drafts problem. A column that somebody maintains by hand is wrong within a week, and a column maintained by a system that cannot see the other side of the publish is wrong permanently.

Add a dashboard. More places showing the same wrong number, faster. A dashboard increases confidence without touching correctness, which is precisely the wrong direction for a figure whose problem is that it is believed.

Reconcile periodically. Right instinct, wrong cadence. A reconcile that runs when somebody remembers tells you the number was wrong for an unknown period, and the interesting question — when did this start — is not answerable afterwards.

The actual mechanism#

Prefer numbers that are derived from the thing itself at the moment of asking, and for the ones that cannot be, say what they are derived from.

derive    state from disk and from the remote, every run, never stored
name      the rule a derived number uses, in the output
check     the rule against a case it would get wrong
pair      every count with the question "what would be missing"

The first line is what fixed the status column: nothing about publication state is remembered anywhere. Each run inspects the files and the remote record and computes the state, so the number is a measurement rather than a memory, and a stale one is impossible by construction.

The second line is what would have caught the backlog count, and it is the cheaper half. The rule was "no file at this name means unwritten" — correct for everything except a post whose name changed, which is a case anybody could have identified in a sentence if the rule had been stated next to the number instead of living inside the tool.

That is the general shape of the remedy and it costs almost nothing. A number printed with its own predicate invites the reader to think of a counterexample, which is the cheapest verification that exists and the one no amount of internal testing substitutes for — the person reading the output usually knows the domain better than the person who wrote the rule.

The sub-case worth separating is the direction of a count's error. Most counting rules fail asymmetrically: a feed under-reports, a file-existence check over-reports what is missing, a search index lags. Knowing which direction a number is wrong in is often more useful than knowing the magnitude, because it tells you whether a decision made on it is safe.

A count that over-reports work remaining is survivable: you do slightly more than necessary and discover the overlap. A count that under-reports it is how something gets shipped as complete. The two feel equally like errors and they are not equally costly, and knowing which one a given rule produces is a property of the rule you can work out once, in advance.

The fix#

compute   from disk and remote, at read time, never from a stored column
state     the derivation rule alongside the number
test      the rule on the case that breaks it, before trusting the output
prefer    counts with a direction of error you can name

Derived at read time from the two sources, with the rule stated in the same breath as the number:

type State = "planned" | "drafting" | "ready" | "remote-draft" | "remote-live";

/** Disk and remote are the sources. There is no stored status column to drift. */
export function stateOf(p: Post, remoteIds: Set<string>): State {
  if (!p.fileExists)                            return "planned";
  if (p.meta.status !== "ready")                return "drafting";
  if (p.meta.remote_url)                        return "remote-live";
  if (p.ledgerId && remoteIds.has(p.ledgerId))  return "remote-draft";
  return "ready";
}

export function tally(posts: Post[], discovery: { ids: Set<string>; complete: boolean }) {
  const rows = posts.map((p) => stateOf(p, discovery.ids));
  const count = (s: State) => rows.filter((r) => r === s).length;
  return {
    counts: Object.fromEntries((["planned","drafting","ready","remote-draft","remote-live"] as const)
      .map((s) => [s, count(s)])),
    // The number travels WITH its derivation rule and its direction of error.
    derivation: "state read from disk + remote each run; no stored column",
    bound: discovery.complete ? "exact" : "LOWER BOUND — remote list incomplete",
  };
}

The same corpus, read two ways:

$ tally                      # remote discovery incomplete
  planned 5 · drafting 9 · ready 86 · remote-draft 0 · remote-live 112
  bound: LOWER BOUND — remote list incomplete

$ tally --verified
  planned 5 · drafting 9 · ready 1 · remote-draft 75 · remote-live 168
  bound: exact

Eighty-five posts move between those two readings, and nothing about the first one looks wrong. The bound line is what stops it being quoted as a total.

The third line is the one people skip because it feels like testing arithmetic. It is not: the arithmetic was always right. What needs testing is the predicate — "a file at this path means written" — and the test is to construct the case it misses and see whether the number moves.

What makes the derived approach affordable is that these are small sets. A few hundred files and a few hundred remote records can be enumerated on every invocation in well under a second, so there is no performance argument for caching the answer — and the cache is the thing that goes wrong.

The temptation to cache arrives later and for a good-sounding reason: somebody wants history. A derived number has no past, so the obvious move is to record it each run — and the moment that record exists, something starts reading it instead of recomputing. The way to keep both is to make the stored copy plainly an archive, written by one path and read by nobody who is making a decision.

How we proved it#

Both wrong numbers are documented with their corrections, which is the only reason this post has specifics rather than a lesson. A hundred stored drafts against roughly sixty live, and a fifty-six-item backlog against a true forty-five. Each discrepancy was found, measured, and the rule that produced it changed.

The second fix is checkable in the way the first was not. The status column was replaced with a computation, which cannot go stale. The backlog rule was extended to graduate a row when a different named post covers it — that is a judgement, recorded per row with a reason, so the count is now derived from the files plus eleven stated decisions rather than from the files alone.

What I cannot tell you is how long each was wrong. Neither number carried a history, so the question "when did this start" has no answer and will not have one. That is the cost of a derived number, and it is a real one — a stored column is wrong and auditable; a derived one is right and has no past.

What I have not solved: the numbers that come from outside. Indexed pages, impressions and referring queries are computed by somebody else from data I cannot see, with rules I can read descriptions of and not verify. Everything above applies to counts I compute; for the external ones the honest position is that I am trusting a derivation on the basis of reputation.

That is not a counsel of despair about external numbers, but it does change what they are for. A figure I cannot verify is useful as a trend and unsafe as a threshold — watching it move is reasonable, and triggering an action when it crosses a line gives decision-making authority to a computation whose rule I have never seen.

And a limitation of preferring durable metrics generally: indexed pages and referring queries move slowly, which is what makes them worth watching and also means they give almost no feedback on a change made this week. The flattering numbers are popular partly because they respond, and a measurement discipline that offers nothing to look at for a quarter is one people abandon.

The compromise I have settled on is to keep a fast number for the feeling and a slow one for the decision, and never to let the fast one into a conclusion. Watching something respond is a legitimate human need; the error is letting the thing that responds be the thing you act on, because responsiveness and significance are close to unrelated.

The thing that reframed it for me was realising I had been applying scepticism in only one direction. I knew to distrust a number because it measured something shallow, and I had no habit at all of distrusting a number because the thing computing it might be wrong — even though the second is far more likely to be load-bearing in a decision, precisely because it looks like a fact rather than a proxy.

What this generalizes to#

A metric has two ways to lie, and the measurement literature is almost entirely about the first one.

The first is measuring the wrong thing: a proxy that correlates with what you care about under some conditions and not others. That failure is well-covered, widely understood, and usually discussed as vanity metrics. The second is measuring the right thing with a broken rule — a count over a set assembled by a predicate nobody examined — and it is invisible, because the output is a number of the expected kind in the expected range.

So the useful habit is to ask, of any figure you are about to act on, what set is this counting and what decides membership. If the answer is a stored column, it is a memory rather than a measurement. If it is a rule, write the rule next to the number — because a rule that is visible gets argued with, and a rule inside a tool gets believed.