A tag feed is not a list of your posts

A platform offers a feed of your posts under a tag. It is public, it is free, it parses cleanly, and it is the obvious way to find out what you have published there.

It returns what the feed decided to include. That is not the same question.

The case this post is about#

One discovery question — what have I already published? — answered two ways:

$ discover                      # no CDP
  method feed · 112 found · complete false
  missingLocally → null  (refusing: lower bound)
$ WEB_DEBUG_CDP=… discover
  method browser · 175 found · complete true
  missingLocally → 0

The feed found 112. There were 175. The feed was not broken: it is capped, it is tag-scoped, and a post with no matching tag or past the cap simply is not in it.

Read as a total, that gap is 63 posts the tool believes do not exist — and the next run creates them again, permanently, because the create has no undo. This post follows those two numbers, because the refusal in the first case is the whole design: a lower bound cannot answer "does this exist".

The symptom#

A post gets created twice. A tool checks whether a piece has already been syndicated, the feed does not contain it, the tool concludes it is missing, and it publishes a second copy — on a platform whose API, in the common case, has no delete endpoint.

The second thing is the direction of the error, which is what makes it dangerous. A feed under-reports: it has a length cap, it may exclude older items, it may omit drafts and unlisted posts entirely, and it reflects an index that updates on its own schedule. So every mistake runs the same way — the feed says less exists than does, and a tool reading it as a total concludes that something is absent when it is present.

The third is that the failure is invisible at small scale and arrives all at once. With twenty posts, the feed's cap is above your total and it behaves exactly like a complete list. The behaviour does not change gradually as you grow past the cap; it changes at the item that falls off the end, and everything older than that becomes permanently invisible to the check.

The fourth is that it looks authoritative. A feed is first-party, served by the platform, and describes your own content — every signal says this is the canonical answer. Nothing in the response says "this is a page of results subject to a cap", because that is obvious to whoever designed the feed and invisible to whoever is consuming it as an inventory.

There is a fifth that is really a property of the whole class: the check fails safe in the wrong direction. A conservative tool, finding no evidence a thing exists, does the cautious thing and creates it — and on an append-only platform the cautious action is the irreversible one. Caution and safety point opposite ways here, which is why the instinct does not protect you.

Why the obvious fix failed#

Page through until the feed is exhausted. Correct where it is possible and often it is not. Many feeds have a hard cap rather than pagination, and when the cap is the design there is no page two — the oldest items are not further down, they are absent.

Cross-check two feeds. More data and the same category of answer. Both are lower bounds, so their union is a better lower bound and still not a total. This feels like verification and is really a bigger sample.

The reason it feels like verification is that two independent sources agreeing normally is evidence. It is not evidence here because the two sources share the failure mode: both truncate, both truncate the oldest first, and both will therefore agree precisely about the items they are jointly missing.

Search the platform for the title. Better, because search reaches further back than most feeds — and search is also an index with its own latency and its own matching rules. A recently created post may not be findable for some window, which is exactly the window during which a duplicate would be created.

Search has a second problem that is easy to overlook: it matches on text, and the thing you are asking about is identity. Two posts with the same title are indistinguishable to a search and completely distinct to the platform, so a positive result is weak evidence in the one case that matters — a republished or renamed piece — and a negative result is weak evidence always.

Assume anything not in the feed is unpublished. The assumption that produced the duplicates. It is reasonable when the feed is complete, unstated when it is written down, and false the moment the corpus outgrows the cap.

The actual mechanism#

Keep your own record of what you published, written at the moment you published it, and treat the platform's feed as corroboration rather than as the source.

ledger    one local record per syndicated post, written immediately
check     the ledger decides whether to create; the feed never does
feed      used to reconcile and to detect drift, not to answer "does it exist"

The second line is the whole fix. The question "has this been published" is a question about an action you took, and the authoritative record of an action you took is one you wrote, not one you infer from a remote index that was designed for readers rather than for inventory.

The sub-case that determines whether the ledger holds is when it is written. The record has to be created as the very next thing after the remote call returns, in the same code path, before anything else can fail. A ledger written at the end of a batch is a ledger missing every entry after the point the batch died — and the window between a successful remote create and a local record is precisely where a permanent duplicate is born.

There is a related use for the feed that is genuinely good, and worth separating from the bad one. Comparing the feed against your ledger finds drift: items you believe you published that the platform does not show, which is a real signal about deletion, moderation or a failed create. Used that way the lower bound is informative; used as the answer it is a duplicate generator.

The fix#

Feed discovery answers "at least these exist", never "these are all". Treating it as a list is how a create phase mints a duplicate of a post it simply could not see:

type Discovery = { items: Post[]; complete: boolean; method: string };

async function discover(): Promise<Discovery> {
  if (Deno.env.get("WEB_DEBUG_CDP")) {
    return { items: await listViaBrowser(), complete: true, method: "browser" };
  }
  // RSS tag feeds are capped and tag-scoped: a post with no matching tag, or past the
  // cap, is absent. That is a LOWER BOUND and must travel as one.
  const items = (await Promise.all(TAGS.map(feed))).flat();
  return { items: dedupe(items), complete: false, method: "feed" };
}

export async function missingLocally(local: Post[]): Promise<Post[] | null> {
  const d = await discover();
  // Refuse to compute a difference from an incomplete set. The answer would be
  // "these are missing", and the true answer is "these, plus anything I cannot see".
  if (!d.complete) return null;
  return local.filter((p) => !d.items.some((r) => r.id === p.id));
}

Both paths, on the same corpus:

$ discover                      # no CDP
  method feed · 112 found · complete false
  missingLocally → null  (refusing: lower bound)

$ WEB_DEBUG_CDP=… discover
  method browser · 175 found · complete true
  missingLocally → 0

Sixty-three posts separate the two numbers. A create phase fed the first one creates sixty-three duplicates, every one of them permanent.

write     the ledger entry as the next statement after the create returns
decide    from the ledger alone — the feed is never the gate
reconcile feed against ledger to find drift, in that direction
never     batch creates against a platform with no delete endpoint

The fourth line is the backstop that makes the rest survivable. If the check is wrong, a serial process creates one duplicate and you find out; a batch creates fifty, and on a platform with no delete endpoint the recovery is fifty manual operations in a web interface.

What makes the ledger cheap is that it is not a database. A file per post, or one append-only file, written locally, in the repository, is enough — the requirement is that it is written synchronously and that nothing else is consulted when deciding whether to create.

Putting it in the repository has a second benefit worth naming: the record travels with the work and is visible in review. A duplicate-prevention mechanism that lives in a developer's home directory protects one machine, and the next person to run the syndication from somewhere else starts with an empty ledger and a full corpus.

The hard part is not the file; it is the discipline that nothing else is allowed to answer the question. Every subsequent convenience — a quick feed check, a search, a glance at the platform — reintroduces the original bug the moment somebody uses it to decide, and they will, because it is right there and it usually works.

How we proved it#

The under-reporting is directly observable. Count the items a tag feed returns and compare against what you know you have published. If the feed's number is suspiciously round — twenty, fifty, a hundred — it is a cap rather than a total, and that single observation settles whether it can be used as an inventory.

The ordering requirement is demonstrable by interruption. Kill a syndication run between the remote create and the local write, then run the check again: the post exists remotely, the ledger does not know, and the next run creates a second one. That is a one-minute experiment and it is the argument for writing the record as the immediate next statement rather than at the end.

What I cannot tell you is how many duplicates existed before I looked. Finding them requires enumerating the platform side exhaustively, which is the capability whose absence caused the problem. I found the ones I found; a corpus-wide figure would need the complete list I did not have.

What I have not solved: reconciling a platform that genuinely has no complete enumeration. If the feed is capped and the API offers no listing, there is no mechanical way to establish what is there — the ledger becomes the only record, and a divergence introduced before the ledger existed stays undetectable. That is the situation for anything published before you started keeping one.

The practical answer to that gap is to enumerate once, by hand, at the moment you adopt the ledger, and accept that the resulting record is a snapshot of belief rather than of fact. It is not satisfying and it is strictly better than starting from nothing, because every entry after it is authoritative and the uncertainty stops growing.

And a limitation of the ledger approach: it records intent and outcome from one side. A post removed by the platform, or by a collaborator in the web interface, leaves the ledger saying something exists that does not — which is the opposite error, and it is why the reconcile direction matters. The ledger decides whether to create; the feed is how you learn it went away.

The thing that reframed it was noticing that I was using a reader-facing surface as an administrative one. A feed exists so somebody can follow your writing, and every design decision in it — the cap, the ordering, the omissions — is right for that purpose and wrong for mine. The mistake was not trusting the data; it was asking a question the interface was never built to answer.

What this generalizes to#

Most listing endpoints return a bounded view, and almost none of them say so, which means the default reading of a successful response is wrong in a specific direction.

The pattern recurs wherever a list stands in for a set. A directory listing truncated by the shell. A search API returning the first page with no total. A log query bounded by a retention window. A webhook history capped at the last hundred events. Each answers "here are some" while being read as "here are all", and because the response is a success, nothing anywhere reports the difference.

So the question worth asking of any list you are about to treat as complete is what would be missing if it were not — and if the honest answer is "the oldest, the quietest, or the ones past a cap I did not know about", then what you are holding is a floor, and any decision that needs a total has to come from somewhere else.