The eval that passed because the function returned a plausible stub

TL;DR#

We scored a code-generating agent with an LLM judge: show the judge the task and the generated code, ask whether the task was accomplished, average the scores. The suite reported 91%.

Then we executed the generated code. Actual pass rate: 54%. The gap was almost entirely functions that looked exactly right — correct signature, plausible body, good names, a sensible-looking return — and did not work.

A judge reading code assesses whether it resembles a correct solution. That is a measure of fluency, and fluency is the one thing a language model is guaranteed to have. If the artifact can be run, run it.

The case this post is about#

An eval harness for an agent that implements functions from a docstring:

JUDGE_PROMPT = """
You are evaluating generated code.

TASK: {task}
GENERATED CODE: {code}

Does the code correctly accomplish the task? Answer PASS or FAIL with one sentence.
"""

def evaluate(task, code) -> bool:
    verdict = judge.complete(JUDGE_PROMPT.format(task=task, code=code))
    return verdict.startswith("PASS")

120 tasks, a judge on a strong model, and a reported 91% pass rate that went into a weekly report for two months.

The symptom#

A user reported that a generated function "returns nothing useful". The function:

def parse_duration(s: str) -> timedelta:
    """Parse strings like '1h30m', '45s', '2d12h' into a timedelta."""
    match = re.match(r'(?:(\d+)d)?(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?', s)
    if not match:
        raise ValueError(f"invalid duration: {s}")
    days, hours, minutes, seconds = match.groups()
    return timedelta(
        days=int(days or 0), hours=int(hours or 0),
        minutes=int(minutes or 0), seconds=int(seconds or 0),
    )

The judge's verdict:

PASS — The function uses a regex to extract day, hour, minute and second
components and constructs a timedelta correctly.

That reasoning is accurate about the structure and the code is broken: re.match with an all-optional pattern matches the empty string at position 0, so parse_duration("garbage") returns timedelta(0) rather than raising. The if not match branch is unreachable.

Then we ran the whole suite against real tests:

                      judge says PASS   actually passes
120 tasks                  109 (91%)        65 (54%)

The 44 disagreements, categorised:

unreachable error handling (like the above)        11
off-by-one at a boundary                            9
plausible-but-wrong library behaviour               8
returned the right type with wrong values           7
would not even import (bad name, missing symbol)    5    ← the judge still passed these
silently caught and swallowed its own exception     4

Five of them did not execute at all. The judge read code that could not run and said it accomplished the task.

Why the obvious fix failed#

"Use a better judge model." Upgrading the judge moved reported pass from 91% to 88% against a real 54%. It made the judge slightly more sceptical and did not give it the ability to execute code — which is the missing capability, not a quality deficit.

"Improve the judge prompt." We added "check edge cases", "consider whether the regex is anchored", "be strict". Reported dropped to 79% and the correlation with actual pass barely moved: the judge now failed a different, largely unrelated set of 20%.

"Ask the judge to explain, then score the explanation." More tokens, same ceiling. A judge reasoning carefully about code it cannot run is still reasoning about appearance.

"Have the judge write test cases and check them mentally." This is the interesting failure: the judge wrote good test cases and then evaluated them incorrectly, because mentally executing re.match against "garbage" is exactly the kind of thing it gets wrong. Having the right test and the wrong answer is worse than having no test.

The actual mechanism#

An LLM judge is a text classifier over text. Given code, the features available to it are lexical and structural: does this have the shape of a correct solution? Are the names right? Is the control flow sensible? Does it use the library you would expect?

A language model's core competence is producing text with correct shape. So:

The generator and the judge share a failure mode. Both are optimising for plausibility. Code that looks right to the generator looks right to the judge, and the errors are correlated — which is the worst possible property in a verifier.

Correlated is worth being precise about, because it is not the same as "sometimes wrong". An independent verifier that is right 80% of the time still catches most of what the generator gets wrong. A correlated one catches almost none of it — the cases it misses are specifically the cases the generator produces, because both are drawing on the same notion of what correct code looks like.

The practical consequence is that adding more judges does not help. Three models voting on plausibility agree with each other on exactly the failures that matter, so the pass rate goes up and the catch rate does not. What breaks the correlation is a verifier with a different basis — executing the code, checking a property, comparing against a known output — because none of those consult plausibility at all.

Execution is a different kind of evidence. assert parse_duration("garbage") raising or not is a fact, decided by the interpreter, with no opinion involved.

So the division is clear:

If the artifact is executable, execute it. Code, SQL, a config, a regex, a diff that must apply.

Use a judge for genuinely subjective qualities — is this error message clear, is this doc readable, is this naming idiomatic — where there is no executable ground truth and a graded opinion is the actual thing you want.

A judge reading code measures resemblance to a correct solution. The generator is optimised for exactly that, so a judge is close to measuring the generator's strength rather than its correctness. Executing is the only cheap source of uncorrelated evidence.

The fix#

Execute first. Judge only what execution cannot decide.

@dataclass
class EvalResult:
    task_id: str
    compiles: bool
    tests_passed: int
    tests_total: int
    judged: dict | None          # only reached if the tests pass

def evaluate(task: Task, code: str) -> EvalResult:
    # 1. does it even load? (5 of our 44 failed here)
    sandbox = Sandbox(timeout=10, net=False, mem_mb=512)
    load = sandbox.run(f"{code}\n", entry=task.symbol)
    if not load.ok:
        return EvalResult(task.id, compiles=False, tests_passed=0,
                          tests_total=len(task.tests), judged=None)

    # 2. behaviour, against hidden tests written BY HAND, including adversarial inputs
    passed = sum(1 for t in task.tests if sandbox.check(t).ok)

    # 3. a judge ONLY for qualities execution cannot see, and only on working code
    judged = None
    if passed == len(task.tests):
        judged = judge_quality(task, code)      # readability, idiom, error-message clarity
    return EvalResult(task.id, True, passed, len(task.tests), judged)

Four decisions:

Hidden, hand-written tests, including adversarial cases. The tests are the specification and the agent never sees them. Every task's test set includes at least one invalid input, one boundary, and one empty case — which is what caught the re.match class:

Task(
    id="parse_duration",
    symbol="parse_duration",
    tests=[
        Check('parse_duration("1h30m")', equals=timedelta(hours=1, minutes=30)),
        Check('parse_duration("45s")',   equals=timedelta(seconds=45)),
        Check('parse_duration("2d12h")', equals=timedelta(days=2, hours=12)),
        Check('parse_duration("garbage")', raises=ValueError),      # ← the unreachable branch
        Check('parse_duration("")',        raises=ValueError),
    ],
)

A sandbox with no network and a timeout. Generated code is untrusted code, and an eval harness that executes it without isolation is a different article's incident.

The judge runs only on code that passes. Its question changes from "is this correct" — which it cannot answer — to "is this good", which it can. The scores stop being a correctness claim.

And the report separates the two numbers, because conflating them is what let 91% stand for two months:

120 tasks
  compiles:        115 / 120   (95.8%)
  behaviour:        65 / 120   (54.2%)   ← the headline number
  quality (of 65):  4.1 / 5 mean         ← judged, and labelled as judged

Then a calibration check, run monthly, which is the thing that would have caught the drift:

# sample N judged verdicts and compare against execution on the SAME artifacts
def calibrate(sample: list[EvalResult]) -> float:
    judged_pass = [r for r in sample if r.judged and r.judged["verdict"] == "PASS"]
    return sum(1 for r in judged_pass if r.tests_passed == r.tests_total) / len(judged_pass)

Any judge you keep needs a measured agreement rate with ground truth, or you do not know what its score means.

How we proved it#

The headline gap, which is the whole finding:

                        judge-only   execution
pass rate reported          91%         54%

Judge agreement with execution, per category — because the judge is not uniformly bad:

category                       judge agreement with execution
code that does not import                 0%   (5/5 wrongly passed)
unreachable error handling               18%
off-by-one at a boundary                 33%
wrong library semantics                  38%
right type, wrong values                 43%
correct code                             96%   ← reliably recognises working code

The last row matters: the judge is good at approving correct code and bad at rejecting plausible-wrong code. Its false-negative rate is low and its false-positive rate is what ruins it — which is exactly the asymmetry you cannot afford in a verifier.

That prompt improvements did not close it:

judge variant                reported   actual   correlation (φ)
baseline                        91%       54%        0.21
+ "be strict"                   79%       54%        0.24
stronger model                  88%       54%        0.26
+ write-then-check tests        83%       54%        0.19

Four attempts, correlation stuck around 0.2. That table is what stopped us iterating on the prompt.

And the effect on the agent's development, which is the actual cost of a bad eval:

two months of agent changes ranked by judge score vs re-ranked by execution
  changes that improved the judge score and hurt actual pass:   3 of 11

Three of eleven shipped "improvements" were regressions. A bad metric does not just misreport; it steers.

One limit worth stating: hand-writing hidden tests for every task is substantial work — about 25 minutes per task for us, including the adversarial cases. That is the real cost of this approach, and it is why the judge was attractive. The honest framing is that the judge was cheap and measuring the wrong thing, and 50 hours of test-writing bought a metric we can trust.

What this generalizes to#

A verifier must have a different failure mode from the generator. Correlated errors are the thing to design against. Where the checker and the maker share an architecture, a training set, or a notion of plausibility, the checker's agreement is not evidence.

Execute whatever can be executed. Code, queries, configs, regexes, diffs, schemas. A judge is for the genuinely subjective residue, and that residue is smaller than it first appears.

Report shape and behaviour as separate numbers. "95% compiles, 54% works" is honest. One blended figure lets the easier number carry the harder one, which is how 91% survived two months of weekly reports.

Calibrate any judge you keep. Measure its agreement with ground truth on a sample, periodically. An uncalibrated subjective score is a number with no units, and it will be read as if it had them.

Related#

• An agent loop without a terminal condition is a budget with no bottom — verifying a completion claim
• The agent fixed the test, not the bug, and the test was right — when the check itself gets gamed
• 163 checks reported green. None of them read the published page. — checks that cannot fail

Keep reading#

This is part of Evals & verification — measuring whether generated work is actually correct, rather than whether it reads well. Follow MobileByteSensei for the rest.