The agent whose only job is what is missing

A pipeline finishes. Every stage reports success, the summary aggregates them, and the result is a green run with a list of what was accomplished.

The question nobody was assigned is what should have been there and is not.

The symptom#

A set of reports that are individually complete and collectively partial. Each worker describes its own work accurately; none of them has any view of the whole, so the absence of a stage, a dimension, or a source is nobody's finding.

The second thing is that absence is structurally harder to see than error. A worker that fails produces output. A worker that was never dispatched produces nothing, and nothing is exactly what every part of the system is prepared to treat as fine. Detecting it requires holding an expectation somewhere, and the individual workers do not have one.

The third is that the summary makes it worse rather than better. Aggregating successful reports produces a confident overall report, and the aggregation step is in the best position to notice a missing component and is almost never written to do so — it is a formatter, not a critic.

The fourth is the kind of gap that survives. Not a crash, not a failure, but a modality that was never run, a claim that went unverified, a source that was listed and never read, a dimension of the problem nobody enumerated. These are precisely the gaps that a thorough-looking report conceals.

That last category is the expensive one, because the report's thoroughness is doing the concealing. A thin report invites questions; a detailed one answers them, including the ones it should not be able to answer.

There is a fifth property that keeps this invisible for a long time: the gaps do not accumulate visibly. A missing stage leaves no artifact to find later, so a pipeline can run this way for months and the only evidence is work that somebody eventually does by hand, having assumed it was never in scope.

Why the obvious fix failed#

Have each worker check its own completeness. Asks a worker about the scope it was given, which is the one thing it cannot evaluate. A worker told to analyse three files will faithfully report on three files and has no basis for wondering about a fourth.

Enumerate everything up front and check it off. Works for the dimensions you thought of, which is the easier half. The gaps that matter are usually ones nobody enumerated — that is why they are gaps — and a checklist is a record of what somebody already knew to look for.

Make the orchestrator verify. Closer, and the orchestrator is the component most invested in the run having gone well. It has the whole picture and the wrong incentive, and in practice it summarises rather than interrogates.

Add more checks. Each new check covers its own class and the class of gap we are talking about is "something nobody has a check for". More checks does not approach it; it is the wrong axis.

The actual mechanism#

Run a pass whose only question is what is missing, with no responsibility for anything that was done.

input     the run's full output — what every stage produced and claimed
question  what is absent? what was not run, not verified, not read?
output    a list of gaps, which becomes the next round of work

The single-question framing is what makes it work. A critic asked to "review the output" will review the output — assess quality, note weaknesses, comment on what is there. A critic asked only about absence spends its whole attention on the thing nothing else in the pipeline is looking at.

The categories are worth naming explicitly in the prompt, because they are the ones that recur: a modality that was not run, a claim that was asserted and never verified, a source that was identified and never read, a dimension of the problem that nobody enumerated. Naming them gives the critic a starting shape without constraining it to those four.

Leaving the categories out entirely is worse than it sounds. An unprompted critic tends to produce absences at the level of topics — whole areas that could have been considered — which are unfalsifiable and uncountable, where the named categories point at specific actions that either happened or did not.

The sub-case that determines usefulness is what happens to the findings. A completeness critic whose output is read and filed produces a list of known gaps, which is marginally better than unknown gaps and much less than what it could be. The findings should become dispatched work — that is the whole point of running the critic before declaring the run finished rather than after.

There is a failure mode worth anticipating: a critic that generates plausible-sounding absences that are not real. "No performance analysis was done" is always true of something, and a critic rewarded for finding gaps will find them. The discipline is to require each gap to name what would have produced it — which stage, which source, which check — so a gap is a concrete missing action rather than a topic.

The requirement has a useful side effect: it makes the critic's output auditable. A grounded gap can be checked in a few seconds by looking at whether that action occurred, so a critic producing bad findings is caught quickly rather than slowly eroding trust in the whole pass.

The fix#

isolate   a dedicated pass with no responsibility for the work itself
ask       only about absence, with the recurring categories named
ground    every gap must name the specific action that did not happen
feed      findings become dispatched work, not a closing section

The categories are the prompt, and the grounding requirement is the return type:

/** The recurring shapes of absence. Named, because "anything missing?" returns nothing. */
const CATEGORIES = [
  "a declared capability that never ran",
  "a claim with no evidence attached",
  "a source that was listed and not read",
  "a target or locale or platform the step covered a subset of",
  "a check that was authored but never executed",
] as const;

/** A gap must name the ACTION that did not happen. Prose about risk does not qualify. */
interface Gap {
  category: typeof CATEGORIES[number];
  action: string;          // the thing to dispatch, phrased as work
  evidence: string;        // why it is believed absent — a path, a count, a missing field
}

function isGrounded(g: Gap): boolean {
  // The two filters that keep this from becoming a worry list. A gap that cannot name an
  // action is an observation, and one with no evidence is a hunch — both read as findings.
  return g.action.trim().length > 0 && /[/.:]|\b\d+\b/.test(g.evidence);
}

async function critique(run: RunRecord): Promise<Gap[]> {
  const raw = await agent(
    `Report ONLY what is absent from this run. You are not reviewing the work that was done
     and must not comment on its quality. For each category below, state either a specific
     gap or "none". Every gap names the action that did not happen.\n\n` +
      CATEGORIES.map((c) => `- ${c}`).join("\n"),
    { schema: GAPS, context: run },
  );

  const grounded = raw.gaps.filter(isGrounded);
  // Dispatched, not appended to a report. A findings section at the end of a green run is
  // read by nobody, which is the whole reason the critic exists as a separate pass.
  for (const g of grounded) await enqueue({ from: "completeness-critic", ...g });
  return grounded;
}

And what it returned on a run that had reported success on every stage:

$ deno task critique --run wf_7c21a4

5 gap(s), all dispatched:
  capability-never-ran   device-test — PIPELINE_STATE shows status:not-run, 0 captures on disk
  claim-without-evidence "renders correctly on all five targets" — 1 target in the run log
  source-listed-not-read CORE_TESTING.md declared in the pack, 0 reads recorded
  subset-coverage        34 of 210 assets rendered — platforms[] read as a media scope
  authored-not-executed  3 *ActionTest.kt written this run, 0 gradle test invocations

  (2 further gaps dropped as ungrounded: "error handling may be incomplete",
   "consider reviewing the migration path")

The two dropped lines are the ones worth noticing. Both are plausible, both would have read fine in a report, and neither names an action — so neither could have been dispatched, which means neither would ever have been done. A critic that keeps them has produced a longer document; one that drops them has produced a queue.

The third line is the one that keeps the output honest, and it also makes the findings directly actionable: a gap expressed as "the third source in the list was never opened" is a task, while "the research could have been deeper" is a mood.

The fourth line changes what the critic is for. Run at the end, it is a disclaimer. Run before the run is allowed to finish, it is a loop — and the loop terminates when a pass produces no grounded gaps, which is a much better stopping condition than anybody's sense that the work is done.

That is also the version I would build first if resources were limited. A critic that runs once at the end and files a list is a report; a critic whose empty output is the run's termination condition is a mechanism, and the difference is entirely in where it sits rather than in what it does.

How we proved it#

The critic finds things, reliably, on runs that reported clean. That is the whole empirical claim and it held on the first attempt: the first pass on a completed, green pipeline produced several grounded gaps, of which the most useful was an entire class of check that had been declared applicable and never dispatched.

The grounding requirement is what separates signal from noise, and I know that because I ran it without the requirement first. The ungrounded version produced a page of reasonable-sounding observations that were mostly unfalsifiable; adding "name the action that did not happen" cut the output by most of its volume and all of its uselessness.

What I have not measured is how many real gaps the critic misses. It has the same blind spot problem as anything else — it can only notice absences it can imagine — and there is no way to measure the gaps that neither the pipeline nor its critic thought of.

What I have not solved: the critic that is too expensive to run often. A thorough completeness pass over a large run costs real money, and running it on every pipeline execution is not obviously worth it. I run it on substantial work and not on small changes, and the boundary between those is a judgement I make badly.

And a limitation worth naming: this finds absences, not errors. A run where every stage was dispatched and every one of them was wrong passes a completeness critic cleanly. It answers one question well, and the question is narrower than "was this any good".

The thing that reframed it was noticing that no component in a normal pipeline has absence as its job, and that this is a structural property rather than an oversight. Workers report on their work; summarisers aggregate reports; checks test what exists. Absence is the one thing that falls between all of them, which is why it needs a component of its own.

That also explains why it works as well as it does despite being a simple idea. The critic is not cleverer than the rest of the pipeline — it is just pointed at the one direction nothing else is pointed at, and in a well-built system that direction has accumulated everything nobody looked at.

What this generalizes to#

A system of components that each report on their own work has no component reporting on the work nobody did.

The pattern is visible well outside agent pipelines: a project where every team's status is green and the integration nobody owns is missing, a test suite where every test passes and the untested path has no representative, a budget where every line is justified and the unfunded necessity has no line. In each case the reporting structure mirrors the work structure, so unassigned work is unreported by construction.

So the question worth asking of any reporting system is who would notice if something were simply absent. If the answer is nobody in particular, then the system's completeness rests on everybody having remembered — and the cheapest correction is one pass whose only job is to ask what is not there.