The agent said 'fixed'. The list said otherwise

"deleted": true

The item was still in the list. It is still in the list now, several minutes later, after a hard reload. The delete reported success and the delete did not happen.

The case this post is about#

One field, in one JSON result:

"deleted": true

That was the agent's report after clicking a delete control. The item was still live, still listed, and still reachable at its own URL.

Nothing lied. The click succeeded, the call returned without throwing, and deleted: true was constructed from the fact that the code reached the line after the click. The claim describes what the agent did, and was read as a claim about what is now true — and those two come apart exactly when a control is disabled, a confirm dialog is waiting, or the mutation needs a second step nobody knew about.

The symptom#

Four of these in one day, and the shape is identical every time:

The last row is the mirror image and just as costly: a false negative, where the work was done and the verification said it was not. Trusting that would have re-run the append and stacked a duplicate.

What unites them is not carelessness. Each was a reasonable implementation where the action reported on itself. The click did not throw, so the click succeeded. The function returned, so the function worked. In every case the thing being asked — did it work? — was answered by the thing that did it.

Why the obvious fix failed#

The obvious fix is to check the return value more carefully, and it cannot work, because the return value is produced by the same code path that failed.

Consider the exit-0 case, which is the purest example:

void (async () => {
  const browser = await open()
  try {
    const result = await doTheWork()
    console.log(JSON.stringify({ ok: true, ...result }))
  } finally {
    await browser.close()
    process.exit(0)        // ← any throw above lands here and exits SUCCESS
  }
})()

There is no catch. A throw skips the log, runs the finally, and exits 0 with no output at all. The caller sees status 0 and reasonably concludes success. Nothing about "check the return value harder" reaches this — the return value is 0, and it is lying.

The subtler version is the scroll loop, where nothing threw and nothing was mis-handled. It collected rows until the count stopped growing, which is a correct-sounding termination condition and happens to coincide with a pagination boundary. The code did exactly what it said. What it said was not what I needed.

The actual mechanism#

The fix is positional, not diligence-based: the check must read the state through a different path than the action wrote it.

For the delete, that means not believing the click and instead asking the system whether the thing exists:

await deleteButton.click()
await page.goto(itemUrl)                    // different path: fetch it fresh
const body = await page.evaluate(() => document.body.innerText)
const gone = /author deleted this|not found/i.test(body)
if (!gone) bail("clicked Delete but the item still renders — treat as NOT deleted")

Note what this does not do: it does not check that the click landed, that the menu opened, or that a confirmation dialog appeared. All of those are properties of the action. The only question is whether the item is gone, asked of the thing that would know.

My first version of exactly this check was wrong in an instructive way. I tested for a non-200 on the item URL, reasoning that a deleted item would not serve one. It serves 200 and renders "The author deleted this story". The status code is a property of the transport; the deletion is a property of the content. I was still checking the wrong layer, one level further out.

The fix#

Three habits, each of which caught something the others did not.

Assert on the outcome, not the operation. After a write, read. After a delete, look. After a create, fetch the created thing. If the system offers no way to read back — some APIs genuinely do not — then say so explicitly rather than treating the write's own response as confirmation.

Cross-check against a number the system publishes. The scroll loop got a hard check for free: the page prints its own total. Requiring scraped == declared turns "I collected some rows" into "I collected all of them, and here is the second source that agrees".

counts: {"published":{"declared":100,"scraped":100}}

That is worth more than any amount of loop review, and it is one line.

Make failure loud and never silent-success. The finally bug is worth fixing generically:

} catch (e) {
  console.error(JSON.stringify({ ok: false, error: (e as Error).message }))
  code = 1
} finally {
  await browser.close().catch(() => {})
  process.exit(code)        // carry the real status out
}

An empty successful exit is the worst possible outcome, because it is indistinguishable from a clean run and strictly less informative than a crash.

How we proved it#

The delete, before and after the read-back was added. Before, the action's own report:

"deleted": true

After, asking the system instead:

clicked Delete but https://…/edit still renders a story — treat as NOT deleted

Same code path for the click. The difference is entirely in what was asked afterwards. And once the real bug was found — a selector matching a heading with the same text as the button below it — the same check confirmed the fix rather than merely failing to complain:

stillInDraftsList: false
editorSays: "The author deleted this Medium story"

Honest limitation: this catches did the thing happen, not was it the right thing. A delete that removes the wrong item passes every check above, because the item I asked about is indeed gone. That is a different claim, and it needs a different assertion — resolve the target by a stable id recorded at creation, never by matching a title, because titles are not unique and they get edited.

When the verifier is the thing that is broken#

The uncomfortable case is a check that fails on work which succeeded, because the instinct it triggers is exactly wrong.

An append step reported the text missing. The text was on the page. The comparison was byte-for-byte, and the editor had applied smart typography — a spaced em dash stored as an unspaced one — so a string a human would call identical failed the test. Everything about the check looked careful; it read the document back, it searched for the exact content it had written, it did not trust the click.

What makes this worse than a false positive is the response it invites. A verification saying "not done" tells the operator to run it again, and the operation it points at is the one least able to survive a second run. A too-strict check on an append is a duplicate generator.

The repair is to compare on what a reader would call the same:

const norm = (s: string) =>
  s.replace(/[‐-―—–]/g, "-").replace(/\s+/g, " ").trim().toLowerCase()

Every transformation the target applies between writing and reading — smart quotes, dash substitution, whitespace collapsing, case folding in a search index — has to be normalised away first, or the check fails on its own success.

The general lesson is that a verifier is code too, and it gets the least review of anything in the system, because when it passes nobody looks and when it fails everyone assumes the subject is at fault. Both of today's verification bugs were in the checking layer, not the thing being checked.

What this generalizes to#

"Fixed" is a hypothesis about the world, and the code that made the change is the least reliable witness to it. Before accepting a claim of done — your own, an agent's, a tool's — ask which path produced the evidence. If it is the same path that did the work, you have a restatement rather than a verification, and the difference only shows up when something has gone wrong.