Stop counting findings and start counting empty rounds

An audit is asked to find problems in a codebase. The natural way to bound it is a number: find ten, or run three passes, or spend a budget.

Every one of those stops for a reason that has nothing to do with whether there are more.

The symptom#

A search that terminates on its own arithmetic. Ten findings were requested, ten were produced, the run reports complete — and the only thing established is that there were at least ten. Whether there were eleven or two hundred is outside what the run could tell you.

The second thing is that the stopping point shapes the findings. A search told to find ten will produce the ten most findable, which are the obvious ones — and the obvious ones are, by construction, the ones most likely to have been noticed already. The number caps the search exactly where it starts becoming useful.

The third is the shape of the output, which looks the same whether the search was thorough or truncated. Ten findings, confidently reported. Nothing in the report distinguishes "there were ten" from "I stopped at ten", and a reader has no way to tell which happened.

The fourth is what a fixed number of passes does instead, which is worse in a subtle way. Three passes over the same material tend to surface overlapping sets — each pass covers similar ground — so the third pass finding nothing new is weak evidence, because it was always likely to look where the first two looked.

That weakness is the thing that makes "run it a few times" feel more rigorous than it is. Repetition without variation measures the stability of one approach, not the completeness of the search.

There is a fifth property that makes all of this harder to notice than it should be: the count-based run is almost always right about what it did find. The findings are real, the report is accurate, and nothing about the experience of reading it suggests an absence. The failure has no texture.

Why the obvious fix failed#

Ask for more. Raise the target to fifty. It finds fifty if fifty exist and otherwise runs out, and now you have a different arbitrary number with the same property: a stop that carries no information about what remains.

Run until the budget is exhausted. Honest about cost and silent about coverage. A run that stopped because it ran out of money has told you about the money.

Keep going until the agent says it is done. Delegates the question to a judgement that is systematically unreliable in this direction — a searcher that has found several things tends to feel finished, and "I think that is all of them" is an impression rather than a measurement.

Deduplicate and count unique findings. Necessary and not a stopping rule. Knowing you have forty distinct findings still does not say whether a forty-first exists.

Stop when the findings get boring. The intuition behind a lot of informal stopping, and it measures the searcher's interest rather than the material. Late findings are often the subtle ones, so "these are getting less interesting" and "these are getting harder to see" feel identical from inside and point in opposite directions.

The actual mechanism#

Stop on consecutive rounds that find nothing new, not on a count.

round     a search pass over the material
new       findings not already in the accumulated set
dry       a round that produced nothing new
stop      after K consecutive dry rounds — K ≥ 2

The arithmetic is different in kind from a target. A count tells you when the searcher stopped producing; consecutive dry rounds tell you something about the material — that repeated independent attempts are no longer turning anything up, which is the closest thing to evidence of exhaustion that a search can give.

K has to be at least two, and the reason is specific: a single dry round is routinely a bad round. A search that happened to look in the wrong places once produces a dry round while plenty remains, and stopping on it is stopping on noise. Requiring two consecutive dry rounds makes that coincidence twice as unlikely, which is cheap insurance.

The sub-case that determines whether this works at all is what the dedup is against. Deduplicate against everything seen, including findings that were investigated and rejected — because if rejected findings are not in the seen set, they get rediscovered every round, every round looks productive, and the loop never converges. This is the single most common way a dry-round loop fails to terminate.

It is worth being concrete about why, because the bug reads as a small bookkeeping choice. A finding that was examined and dismissed is still a finding the search produced; if the set of seen items only contains the ones that survived, then every round re-surfaces the dismissed ones, every round has new items by its own definition, and the dryness condition can never be satisfied.

And rounds have to vary. Identical rounds converge on the first thing they all miss: if every pass uses the same approach, dryness means that approach is exhausted, which is not the same as the material being exhausted. Varying the angle each round — different search strategy, different starting point, different question — is what makes a dry round mean something.

The variation does not have to be sophisticated. Changing where a pass starts, which dimension it looks along, or what question it is asked produces materially different coverage — and in my experience the crudest variations work nearly as well as carefully designed ones, because what matters is that the misses are uncorrelated rather than that each pass is optimal.

The fix#

accumulate   one set of everything seen, including rejected findings
vary         each round approaches differently; identical rounds prove little
dry          count consecutive rounds producing nothing new
stop         at K consecutive dry rounds, and report K and the round count

As code, with the four lines above in the order they have to run:

const seen = new Set(), confirmed = []
let dry = 0, round = 0

while (dry < K && round < ROUND_CAP) {
  round++

  // VARY. Each finder is asked a different question, so their misses are uncorrelated.
  // Identical rounds measure the stability of one approach, not the completeness of the search.
  const found = (await parallel(FINDERS.map((f) => () =>
    agent(f.prompt, { phase: `find:${round}`, schema: FINDINGS })))
  ).filter(Boolean).flatMap((r) => r.findings)

  const fresh = found.filter((x) => !seen.has(key(x)))

  if (!fresh.length) { dry++; continue }   // a DRY round
  dry = 0                                  // any fresh finding resets the counter

  // The single line this whole mechanism turns on. `seen` is added to BEFORE judging, so a
  // finding that gets rejected below is still never rediscovered. Deduplicating against
  // `confirmed` instead is the non-termination bug: rejected findings come back every round,
  // every round looks productive, and `dry` never reaches K.
  fresh.forEach((x) => seen.add(key(x)))

  const verdicts = await parallel(fresh.map((x) => () =>
    agent(`Try to refute: ${x.claim}`, { phase: `verify:${round}`, schema: VERDICT })))
  confirmed.push(...fresh.filter((_, i) => verdicts[i] && !verdicts[i].refuted))
}

// CONVERGED and CAPPED are different outcomes and the report must not flatten them.
return { confirmed, round, dry, outcome: dry >= K ? "converged" : "round-cap-reached" }

Two details in there are worth more than the loop. dry = 0 on any fresh finding is what makes the dry rounds consecutive rather than cumulative — a version that never resets converges on the total number of unproductive rounds, which is a different and much weaker claim. And outcome is returned rather than inferred, because a caller that receives only confirmed has no way to tell a search that finished from one that ran out of rounds, which is the ambiguity the whole mechanism exists to remove.

The fourth line's reporting half matters as much as the rule. A run that converged after nine rounds with two dry is a different claim from one that stopped at the round cap with findings still arriving, and the output has to distinguish them — otherwise this becomes another stopping rule whose result looks identical whatever happened.

There does still need to be a cap, because convergence is not guaranteed and a loop that will not terminate is worse than one that stops early. The cap is a safety net rather than the mechanism, and hitting it is a reportable outcome: the run did not converge, which is information.

The distinction between converged and capped is the one thing I would not let a summary flatten. They are different states with different follow-ups, and a report that says "complete" for both is reintroducing exactly the ambiguity the mechanism was built to remove.

How we proved it#

The tail is observable directly. Running past the point where a count-based search would have stopped keeps producing findings for several more rounds, and the ones that arrive late are qualitatively different — subtler, more interesting, and not ones anybody had noticed. That is the whole argument for the approach and it is visible in a single long run.

The non-convergence failure is reproducible. Deduplicating against confirmed findings rather than against everything seen produces a loop that runs forever, rediscovering and re-rejecting the same items with every round looking productive. It is an obvious bug once seen and it is very easy to write.

What I have not measured is the right value of K. Two has worked; I do not have evidence that three is better or that two is sufficient in general, and the choice interacts with how much the rounds vary, which I also have not quantified.

The cost is the part people push back on, and the pushback is reasonable — two rounds that produce nothing are two rounds that are paid for. The counter is that they are the only rounds whose result is evidence, which does not make them cheaper, just load-bearing.

What I have not solved: the systematic blind spot. If every round, however varied, shares an assumption, the loop converges confidently on an incomplete set — dryness establishes that the approaches in use are exhausted, not that the material is. Varying the rounds reduces this and does not eliminate it.

And a cost worth naming: this is more expensive than a fixed count, necessarily. The last two rounds are the ones that produce nothing, and paying for them is the entire mechanism — a version that skips the confirming dry rounds to save money has removed the only part that was evidence.

The thing that reframed it was noticing that a count is a statement about the searcher and dryness is a statement about the material. Almost every stopping rule people reach for first — findings, passes, time, budget — describes the process; the question being asked is about the thing being searched.

Which is why the dry round feels wasteful and is not. The round that finds nothing is the only round that produces information about what remains, and it is the one every efficiency instinct wants to cut.

What this generalizes to#

A search for an unknown quantity cannot be bounded by a quantity.

The same structure applies well beyond code audits: looking for security issues, finding all the places a pattern occurs, collecting requirements from stakeholders, chasing down the causes of an outage. In each case the natural bound is effort and the useful bound is diminishing returns, measured by trying again and finding nothing.

So the question to ask of any discovery process is what made it stop. If the answer is a number somebody chose, the result is a lower bound wearing the clothes of a total — and the difference between those two is exactly the part nobody has looked at.