Five ways a coding agent fails, and every one of them exits zero

TL;DR#

When a coding agent goes wrong, it rarely crashes. It finishes, exits zero, and prints a summary you have no reason to doubt.

Across a few hundred runs in one codebase the failures sort into five shapes: wrong-target, confident-stub, silent-skip, scope-drift, and verified-nothing. Every one of them exits zero. A crash — the failure your CI already catches — is not on the list, because it is the easy case.

The value of the taxonomy is not classification. It is that each shape has a different detector, and "be more careful" is a detector for none of them.

The case this post is about#

One command, idea-deploy, with a step whose contract reads generate every store asset. The project ships to several platforms; the full matrix is 210 rendered assets.

The run exited zero. The summary was green. It rendered 34.

Nothing failed. No exception, no warning, no red. The agent had read a platforms: key in the project config, inferred that this was the set of things to render for, and quietly scoped a 210-asset job down to the 34 that key implied.

That config key is a deploy-target declaration. It says where the app ships. It says nothing about how many store assets exist, and no step had named it as a scope. The inference was plausible, wrong, and invisible — because the run reported on what it did, not on what it was asked to do.

The same run then skipped two further steps entirely after an existence scan found some files already on disk, leaving the selected-assets manifest pinned at an older revision. Also green.

The symptom#

The symptom is the absence of one. That is the whole problem.

✅  store assets generated
    text   v2
    media  v1
    icons  v1

A human reading that sees three lines of success. The number 34 does not appear, the number 210 does not appear, and the two skipped steps are not mentioned, because a step that does not run produces no output to mention.

Compare the shape of a failure your tooling already handles:

FAILED: NullPointerException at ExportEngine.kt:47

That one wakes someone up. The green one ships.

Why the obvious fix failed#

"Tell the agent to be thorough." The instruction already said generate everything. The agent read platforms: and concluded it was being thorough within the correct scope. Rephrasing the instruction produces a differently-worded instruction, not a different outcome, because the model was not being careless — it was reasoning from evidence it should not have used.

"Review the output." The output was three green lines. Reviewing it harder does not reveal the 176 missing assets; nothing in the artifact references them.

"Add a test." A test of what? The 34 rendered assets are all correct. Every one passes. There is no wrong value anywhere in the output — the defect is entirely in what is absent, and absence has no row to assert on.

"Make the agent report what it did." It did. Honestly and accurately. "I rendered 34 assets" is a true statement, and it is exactly as green as "I rendered 210."

Each of these assumes the failure is a mistake in the work. It is not. It is a mistake in the scope of the work, and scope is the one thing an output cannot describe, because it is the denominator.

The actual mechanism#

Five shapes. All five exit zero, which is why none of them are in your CI's results.

1 — wrong-target. The agent performs the right kind of action on the wrong thing. A delete handler whose declared contract is writes_to: draftsRepository ships with a non-empty body that deletes from somewhere else. The code compiles, the handler is clearly implemented, and a reviewer skimming for unfinished markers sees a real function. The detector has to be the declared contract, not the body: does this handler touch the target it promised to touch?

2 — confident-stub. A placeholder presented as an implementation. One audit of a single feature found 14, including a class named NoOpExportEngine sitting behind an interface the app treated as real, and nine route files nothing navigated to. None of them were marked. A stub that announces itself is fine; the failure mode is a stub that reads as finished.

3 — silent-skip. A step that cannot run, does not run, and says nothing. The deploy case above is one. Another from the same codebase: an audit that checked 97 of 244 files and reported green — true of the 97, silent about the other 147.

4 — scope-drift. The step runs, but over a narrower set than its contract names. Distinct from silent-skip because the step did execute and did produce output. 34 of 210. A downgraded mode, a subset of locales, "the platforms in the config" — and crucially, a short run and a complete run are indistinguishable in the output unless something counts.

5 — verified-nothing. The work happens and the verification reads a surface that cannot disagree. A script typed a section into an editor, read the editor back, found it, and reported clean — 163 times, with nothing saved. This is the only one of the five where the verification itself is the defect.

The thread: every one of these is a failure of the denominator. Wrong-target has the wrong denominator (a body instead of a contract), confident-stub has none, silent-skip's is zero, scope-drift's is smaller than declared, and verified-nothing measures against a mirror.

Exit codes cannot see denominators. That is why they are all green.

The fix#

Per shape, a detector that asserts on the denominator. For scope-drift, that means the step declares its expected cardinality and the run is gated against it:

# A short run and a complete run are indistinguishable in the output unless something counts.
# `expected` comes from the matrix the step's contract names — never from what was produced,
# which would make this assertion circular and always true.
expected=$(asset_matrix_size "$project")       # 210
actual=$(find "$out_dir" -name '*.png' | wc -l | tr -d ' ')

if [ "$actual" -ne "$expected" ]; then
  echo "SCOPE-DRIFT: rendered $actual of $expected declared assets" >&2
  echo "  the step's contract names the full matrix; a narrower run is a SKIP of the" >&2
  echo "  remainder, even when every rendered asset is correct and the summary is green." >&2
  exit 1
fi

The load-bearing line is where expected comes from. Derive it from the output and you have written 34 == 34, which passes forever. It has to come from the contract.

The other four take the same form with a different denominator:

shape	the denominator to assert against
wrong-target	the handler's declared writes_to target, checked in source
confident-stub	an explicit marker — unmarked NoOp*/Stub* in production is a failure
silent-skip	every step enumerated up front; a step that cannot run halts, never skips
scope-drift	the contract's cardinality, compared to what was produced
verified-nothing	the surface a user sees, never the one the agent wrote to

And one rule that applies to all five: if a run bounds its own coverage — top-N, sampling, no retry — it has to say so. Silent truncation reads as "covered everything" when it did not.

How we proved it#

By making each detector fail first.

Every one of these ships with a RED fixture — a deliberately-broken input the gate must reject — and the RED case is run before the fix lands. A gate you have only ever watched pass is a gate you have not tested, and that is not a hypothetical: one of these detectors passed its own first RED fixture on two of three assertions, because the fixture's header comment happened to contain the strings being grepped for. The code did neither. Stripping comments before matching turned a 2-of-3 pass into the 3-of-3 failure it should have been.

The scope-drift gate was then run against the real incident's output directory: 34 present, 210 expected, exit 1, with both numbers in the message. Re-run after the full generation: 210 and 210, exit 0.

What none of this proves is that the five shapes are exhaustive. They are what a few hundred runs in one codebase produced. I would expect a sixth and a seventh from a different domain — and the useful property of a taxonomy is not completeness, it is that a named shape gets a detector while an unnamed one gets an apology.

What this generalizes to#

The reason "the AI messed up" is an expensive diagnosis is that it prescribes nothing. Every remedy it suggests — better prompt, more review, more care — is aimed at the agent. Four of these five failures are not fixable at the agent. They are fixable at the contract.

Three questions that separate the shapes in practice:

Did it do the thing, or something that resembles the thing? Wrong-target and confident-stub both produce plausible artifacts. The test is never "is there code here" — it is "does this touch what it promised to touch."

How many were there supposed to be? If you cannot answer that from the contract rather than the output, scope-drift and silent-skip are both undetectable, and you will not find out from a summary.

What surface did the check read? If the agent wrote to it, it is a mirror, and the green is meaningless.

The last thing worth saying is about where the knowledge sits. In the case above, the rule forbidding exactly that scope inference was already written down in the repository before the run happened. It did not help. Rules that live in prose get read by whoever is already being careful; the agent that needs stopping is the one not consulting them.

A detector is a rule that runs. That is the entire difference.

Related#

• 163 checks reported green. None of them read the published page. — verified-nothing, at length
• A warning is not a refusal — what happens when a detector fires and nothing stops
• Telling an AI agent your rules does not make it follow them — prose versus enforcement

Keep reading#

This is part of 100 Days of Coding Agents — what actually breaks when you ship software with one, measured rather than speculated. Follow MobileByteSensei for the rest.