Write down what you did to an external service

Most duplicate guards work by asking the remote service whether the thing already exists.

This one cannot, because the service will not answer. Create returns 201; every read returns 404 or 405. The question is this already there? has no remote implementation.

The case this post is about#

One API's four verbs:

create   201
read     405
update   404
delete   404

Create works. Nothing else does — there is no way to read back what you made, no way to change it, no way to remove it, and no way to list what exists.

So the local record is not a cache. A cache is a faster copy of something you could go and ask for; there is nobody to ask. If the local file says a post does not exist, that is not a cache miss to be resolved against the server — it is the answer, and the duplicate guard runs on it.

This post follows those four status codes, because every property the ledger needs — refusing to overwrite, being written as the next statement after the create, being the thing inventory is computed from — follows from the second row.

The symptom#

An agent creating things on a third-party service will, eventually, try to create one twice. The usual reasons are mundane: an ambiguous failure, a retry, a re-run of a batch someone thought had died.

The standard defence is a read before the write. Check whether it exists; skip if it does. That requires the service to support a read, and the surface measured here does not:

create   201
read     405
update   404
delete   404

So the check cannot be performed where the state actually lives. And the consequence of getting it wrong is not recoverable by the agent either — there is no delete, so a duplicate is a permanent public artifact that a person has to remove by hand.

Why the obvious fix failed#

The obvious fix is to search for it. Most services that lack a read-by-id still offer a listing — fetch the collection, look for a match.

That worked until it did not, in a way worth recording. A listing sweep returned 17 items where a later, more patient sweep found 100. The early one had stopped at a pagination boundary, the count looked plausible, and a duplicate guard trusting it would have concluded that 83 existing items did not exist.

The second attempt is to match by title. Titles are not unique, they get edited after creation, and the service normalises typography — so a string that was written one way is stored another, and the comparison fails on a match a human would call identical.

The actual mechanism#

Write a local record at the moment of creation, and treat it as the only authority.

if (await exists(`.publish-out/${slug}.ledger.yaml`)) {
  console.error(`${slug} already has a ledger — it exists remotely. Refusing.`)
  Deno.exit(1)
}

That inverts the usual relationship. A local record is normally a cache — a convenience copy of something authoritative elsewhere, requiring invalidation. Here there is no elsewhere to read, so the record is not a cache of remote state; it is the only representation of it that can be consulted.

Which means the record's integrity is the guard's integrity. If it is wrong, nothing else knows.

The fix#

One detail carries almost all the weight, and it is an ordering.

The ledger write is the very next statement after the create returns. Before logging, before updating any status, before moving to the next item, before anything cosmetic.

The reason is a window. Between the remote create succeeding and the local record landing there is an interval in which the thing exists remotely and nothing local knows. A crash there produces exactly the state the guard is meant to prevent: a created item with no record, and a next run that creates a second.

That window cannot be closed — there is no transaction spanning a third-party API and a local filesystem. What you can do is make it small and make its consequences bounded:

• the ledger write is the next statement, so the window is microseconds rather than seconds
• a failed ledger write is a hard error naming the item, never a warning
• batches run serially, so at most one item can be in that window at a time

That last one is the argument against parallelism I find most convincing. Ten parallel creates means ten simultaneous windows and a crash loses an unknown subset. Serial means the blast radius is exactly one, and you know which.

What makes the ledger the only copy, rather than a cache, shows up in what it has to answer — and in what it has to refuse to answer:

type Entry = { slug: string; remoteId: string; createdAt: string; title: string };

class Ledger {
  constructor(private dir: string) {}

  // No read-back endpoint exists, so this is not "check the cache, then ask the
  // server". There is nobody to ask. Absence here means absence, full stop.
  async get(slug: string): Promise<Entry | null> {
    try { return parseYaml(await Deno.readTextFile(`${this.dir}/${slug}.ledger.yaml`)) as Entry; }
    catch { return null; }
  }

  async record(e: Entry): Promise<void> {
    const path = `${this.dir}/${e.slug}.ledger.yaml`;
    // Refuse to overwrite. A second entry for one slug means a duplicate was
    // created remotely, and silently replacing the first loses the first id.
    if (await this.get(e.slug)) {
      throw new Error(`${e.slug}: ledger already exists — refusing to overwrite`);
    }
    await Deno.writeTextFile(path, toYaml(e));
  }

  // "What have I published?" is answerable ONLY from here.
  async inventory(): Promise<{ count: number; entries: Entry[] }> {
    const entries: Entry[] = [];
    for await (const f of Deno.readDir(this.dir)) {
      if (f.name.endsWith(".ledger.yaml")) entries.push(parseYaml(await Deno.readTextFile(`${this.dir}/${f.name}`)) as Entry);
    }
    entries.sort((a, b) => a.createdAt.localeCompare(b.createdAt));
    return { count: entries.length, entries };
  }
}

The three questions it gets asked, in one session:

const l = new Ledger(".publish-out");
console.log(await l.get("three-phases"));
console.log(await l.get("what-nano-cannot-do"));
await l.record({ slug: "three-phases", remoteId: "7c04b912", createdAt: now(), title: "Three phases" });
console.log((await l.inventory()).count);

{ slug: "three-phases", remoteId: "9f2c1b05", createdAt: "2026-10-03T11:42:08Z", title: "Three phases" }
null
Error: three-phases: ledger already exists — refusing to overwrite
77

The second line is the one that distinguishes a ledger from a cache. A cache returning null means "ask the server"; here it means the item does not exist remotely, and that inference is load-bearing because it is what the duplicate guard runs on. The third line is the guard firing on a re-run that would otherwise have replaced 9f2c1b05 with a second id — and once overwritten, the first id is unrecoverable, because there is no list endpoint to go and find it again.

How we proved it#

The guard, against a slug whose ledger already exists:

already has a ledger at .publish-out/<slug>.ledger.yaml — it exists remotely. Refusing.

That is the whole mechanism working: no network call, no listing sweep, no title matching. A file exists, therefore the thing exists.

The more interesting evidence is what the ledger became without anyone designing it that way.

The status board for this blog reports how many posts are on the remote service. It derives that from ledger presence, not from any field in the posts themselves — because the posts' own frontmatter does not reliably carry the remote URL, and the service cannot be asked.

So a record that started as a duplicate guard is now the answer to "what is the state of the world over there". It was never promoted to that; it simply turned out to be the only thing that knew.

What the ledger does not know#

Two limits, and both follow from the same property.

It knows what you did, not what is true. If someone deletes a post through the web interface, the ledger still says it exists and nothing will correct it. The record tracks your actions, and it diverges from reality the moment anything else acts.

A missing record is ambiguous. No ledger means either the item was never created or the write was lost in that window. Those need opposite responses — create it, or go and look by hand — and the absence cannot distinguish them.

I have no fix for either. What helps slightly is that the second case is loud: a failed ledger write is reported as a hard error naming the slug, so the ambiguity comes with a pointer to the one item that might be affected rather than leaving you to audit everything.

When you do have a read-back#

To be clear about scope: almost none of this applies when the service will tell you what exists.

With a read-back, the remote state is authoritative, your local record is an optimisation, and the correct instinct is the standard one — check before writing, treat any local copy as a cache, and invalidate freely. Building a ledger-as-truth on top of a service that supports reads is reimplementing a database badly.

The test for which world you are in is narrow: can you ask the service whether a specific thing exists, and trust the answer? If yes, ask it. If the answer is a listing that might be truncated, or a match on a mutable field, or nothing at all, you are in the other world and the local record is load-bearing rather than convenient.

What this generalizes to#

Before writing to an external service, find out what it will tell you afterwards. If it can confirm what exists, use that and keep your local state as a cache. If it cannot, write your own record and understand that you have just made it the only source of truth for that state.

Then treat the ordering as the design: the record goes down as the immediate next statement, a failed record write is an error rather than a warning, and the work stays serial so the uncertain window holds exactly one item. Everything else about the guard is easy; that one line's position is what makes it work.