An evaluator with two outcomes will pick one

An evaluator is asked whether an implementation is real behaviour or a stub. It returns a verdict and a justification, and both are well formed.

The evidence it was given could not have distinguished the two.

The symptom#

A confident verdict on an undecidable case. The justification reads well — it cites specifics, it reasons from them — and the reasoning is doing something other than what it appears to be, because the specifics were compatible with either answer.

The second thing is that this is not an error anywhere. A judge asked a binary question answers the binary question; producing "I cannot tell" was never one of the options, so a judge that is functioning perfectly will still produce a guess when the evidence runs out.

The third is the direction of the guess, which is not random. A judge with insufficient evidence tends toward the answer that requires less: passing is usually the less disruptive verdict, it generates no follow-up, and nothing about it will be questioned. The undecidable cases quietly become passes.

The fourth is that the confidence is uniform. A verdict produced from strong evidence and one produced from nothing look identical in the output — same shape, same tone, same structure of justification — so a reader cannot weight them differently even if they wanted to.

That uniformity is what makes the aggregate untrustworthy rather than merely noisy. A set of verdicts where some proportion are guesses, and nothing marks which, is not a slightly degraded version of a good set; it is a set with an unknown amount of fiction in it.

The fifth property is that nothing about the judge needs to be weak for this to happen. A judge with excellent reasoning on decidable cases produces exactly the same confident output on undecidable ones, because the limiting factor is the evidence and not the reasoning. Improving the judge does not move this at all.

Why the obvious fix failed#

Ask for a confidence score. Reasonable, and it produces a number that is itself a guess. A judge that cannot tell whether something is a stub is not well placed to quantify how much it cannot tell, and the scores cluster in a narrow band regardless.

Give the judge more evidence. The right move when it is available, and it does not address the cases where the evidence genuinely does not exist. Some questions cannot be answered from an artifact, and no amount of additional context changes that.

Use several judges and take the majority. Helps with variance and not with this. Judges given the same insufficient evidence tend to guess in the same direction, for the same reasons, so the majority is as wrong as any individual and much more convincing.

Treat low-confidence verdicts as failures. Safer, and it conflates two different things. "This is a stub" and "I could not establish whether this is a stub" lead to different actions — one is a bug to fix, the other is evidence to gather — and collapsing them sends people to look for a defect that may not exist.

It also degrades in a predictable way. People sent to look for defects that are not there learn to discount the failures, and the discount applies to the real ones too — so a safety measure taken to avoid missing bugs ends up reducing how seriously any failure is taken.

The actual mechanism#

Give the verdict vocabulary a third value that blocks but does not accuse.

pass      the evidence establishes the thing is as claimed
fail      the evidence establishes it is not
unclear   the evidence does not establish either — blocks, and asks for more

The property that makes unclear useful rather than a cop-out is that it blocks like a failure while pointing somewhere else. It does not let a release through, so it cannot become the quiet default; and it does not send anybody looking for a defect, because it is a statement about the evidence rather than about the artifact.

Getting this right requires one thing in the prompt that is easy to leave out: unclear has to be described as a correct answer rather than an escape. A judge told it may say unclear "if necessary" will treat it as a failure of its own and avoid it; a judge told that unclear is the right answer whenever the evidence does not distinguish the cases will use it accurately.

The phrasing matters more here than anywhere else in the prompt. "You may answer unclear if you are unsure" invites a self-assessment of confidence; "answer unclear when the evidence does not distinguish the cases" asks about the evidence, which is a question with a determinate answer.

The sub-case that determines whether the whole thing works is what happens to an unclear verdict downstream. If it is treated as a soft failure and ignored, the vocabulary has three values and the system has two. The unclear verdict has to have an owner and a next action — gather this specific evidence, run this check, look at this manually — or it decays into a warning.

This is the failure I would expect most teams to hit, because adding a verdict is a small change and wiring its consequences is not. A three-valued judge in front of a two-valued pipeline produces exactly the old behaviour with extra bookkeeping, and it is easy to ship the first half and call it done.

There is a second-order benefit that took me a while to see: the rate of unclear verdicts is a measurement of the evidence pipeline. A judge returning unclear on a third of its cases is not a bad judge; it is a correct judge reporting that the inputs are inadequate, and that is information the binary version was destroying.

The fix#

vocabulary  three verdicts, with unclear defined as a correct outcome
block       unclear prevents progress exactly as a failure does
route       unclear produces a specific request for evidence, not a warning
measure     track the unclear rate; it describes the inputs, not the judge

The vocabulary and its consequences, which have to ship together or the second half never does:

type Verdict =
  | { kind: "pass" }
  | { kind: "fail"; defect: string }
  | { kind: "unclear"; missing: Evidence };   // names the INPUT, not a feeling

/** The unclear branch must name something obtainable, so the type makes it mandatory. */
type Evidence =
  | { need: "runtime-capture"; of: string }
  | { need: "test-execution"; target: string }
  | { need: "human-read"; file: string; line: number };

/** This wording is the prompt, verbatim, and it is the part that gets softened. */
const RUBRIC = `
Answer "unclear" when the evidence does not distinguish the cases. That is the CORRECT
answer in that situation, not a failure to decide — and it requires naming the specific
evidence that would distinguish them.
Do NOT answer "unclear" because you are unsure. Confidence is not the question.
`;

/** `unclear` blocks exactly as `fail` does. This is the line that is usually missing. */
function gate(v: Verdict): { proceed: boolean; action: string } {
  switch (v.kind) {
    case "pass":
      return { proceed: true, action: "" };
    case "fail":
      return { proceed: false, action: `fix: ${v.defect}` };
    case "unclear":
      // NOT a warning, and not a soft failure. It has an owner and a next action, which is
      // the difference between three verdicts and two verdicts with extra bookkeeping.
      return { proceed: false, action: describe(v.missing) };
  }
}

function describe(e: Evidence): string {
  switch (e.need) {
    case "runtime-capture": return `capture ${e.of} on a device after force-stop`;
    case "test-execution":  return `run ./gradlew ${e.target} and attach the result`;
    case "human-read":      return `read ${e.file}:${e.line} and rule manually`;
  }
}

And the rate, which is the measurement the binary version was destroying:

$ deno task judge --summarise --last 200

pass     118   59%
fail      24   12%
unclear   58   29%
             └─ runtime-capture  41   (no device in the drive that produced these)
                test-execution   13   (the module's tests were never run)
                human-read        4

$ # and the same 200 cases through the two-valued judge, for comparison
pass     171   86%
fail      29   14%

Fifty-three of the fifty-eight unclear cases came back as pass from the binary judge. That is the number worth sitting with: 29% of the pipeline had no evidence either way, and the two-valued version resolved almost all of it in the direction that let a release through. The 41 runtime-capture rows are also not a judge problem — they are one missing device, reported.

The third line is the operational half. An unclear verdict that says "could not determine" is only marginally better than a guess; one that says "could not determine — no runtime evidence was available for this handler" names the missing input, and the missing input is usually obtainable.

The fourth line is the part I would not have predicted was valuable. Watching the unclear rate move tells you whether changes to the evidence-gathering are working, which is otherwise very hard to measure — in the binary world, improving the evidence shows up as verdicts changing, and you cannot tell improvement from drift.

It is also a rate with a sensible target that is not zero. Some questions genuinely cannot be answered from any evidence the pipeline could reasonably gather, so a small standing unclear rate is the correct steady state — and a rate that falls to zero is more likely to mean the third verdict has stopped being used than that the evidence became perfect.

How we proved it#

The undecidable case is easy to construct. Give a judge an artifact where the distinguishing evidence is absent and see what it says. It produces a verdict, with reasoning, and the reasoning is the interesting part — it is usually an inference from style or plausibility, which is exactly what it should not be doing.

The direction of the guess is observable by constructing the same case in both directions. In ours the guesses leaned toward passing, which is consistent with the structural argument and is a small sample; I would not claim it as a general law, only as a reason to assume the lean exists rather than to assume symmetry.

What I have not measured is accuracy on the cases where the evidence is sufficient. The whole discussion above is about the undecidable tail; whether the judge is any good on the decidable part is a separate question and I have treated it as a given, which it is not.

What I have not solved: the judge that says unclear too readily. The failure mode in the other direction is a verdict vocabulary where the third option absorbs everything difficult, and the evaluator stops producing useful signal while remaining technically honest. I have not found a principled way to calibrate this beyond looking at examples.

And a limitation worth naming: three verdicts is not obviously the right number either. The same argument that produces unclear could produce finer distinctions — evidence missing, evidence contradictory, question malformed — and I have not explored whether those earn their complexity.

The thing that reframed it was noticing that forcing a verdict converts a question about evidence into a question about the artifact. The judge has useful information — that it cannot tell — and the output format has nowhere to put it, so the information is discarded and replaced with a guess.

Which is a general hazard of designing output schemas before understanding what the producer actually knows. The schema looks like a neutral container and it is a constraint on what can be expressed, and whatever it cannot express gets rounded to something it can.

What this generalizes to#

A forced choice produces an answer whether or not one is warranted, and the answer is indistinguishable from an informed one.

The shape recurs well beyond evaluators: a survey without a "don't know" option, a triage system with no "needs investigation" state, a classifier whose label set has no "other", a status field that must be green or red. In each case the missing option does not prevent the situation from arising — it only prevents it from being reported.

So the question to ask of any decision-making component is what does it do when it does not know. If the answer is "it decides anyway", then some fraction of its output is fabrication, and nothing downstream can tell which fraction.