Stop parsing your AI's prose. Make it return a schema you can assert on.

A conversion step reported this:

{"frames": 48, "durationMs": 4000, "bytes": 97226, "moving": true}

Four facts I could act on immediately. Had it returned "converted the animation successfully" I would have had one claim I could neither check nor use, made by the only party with an interest in it.

TL;DR#

• Validating a model's answer after you receive it is too late; move the contract to the TOOL-CALL layer so the shape is enforced before the result reaches you.
• A hopeful string becomes something a caller can assert on — and the model retries on mismatch rather than you writing a parser.
• A well-typed wrong answer is still wrong: the schema buys you shape, never correctness.

The case this post is about#

One result from one dispatched worker:

{"frames": 48, "durationMs": 4000, "bytes": 97226, "moving": true}

That is what the caller wanted. What the caller actually received, across a hundred invocations, was that object sometimes — and otherwise the same content wrapped in prose, or with moving as the string "true", or with an extra field, or with one missing.

So every call site grew a tolerant parser: a regex to find the JSON in the prose, a coercion for the boolean, a default for the absent field. Each of those is a small reasonable accommodation, and together they mean no caller can assert anything.

This post follows that one shape, because enforcing it at the tool layer rather than parsing it at the call site is the difference between a contract and a hope — and because a well-typed wrong answer is still wrong.

The symptom#

An agent asked for a result gives you a paragraph. The paragraph usually contains the answer, along with some reasoning, a caveat, and a summary of what it did — and the caller now has a parsing problem that gets solved with a regular expression and an apology.

The failure is not the parsing. It is that prose has no failure mode. A response missing the field you needed reads exactly like one containing it, because both are fluent. There is no mismatch to detect; there is only a value you did not find, discovered later, somewhere else.

Three shapes I have written and regretted:

• a step whose "result" was a sentence, so the caller string-matched on "success" — and matched the word inside "did not succeed"
• a count reported in words, which meant the caller had to decide whether "a few" was ≥ 3
• a list returned as a bulleted paragraph, silently truncated by length, with no way to tell

Why the obvious fix failed#

The obvious fix is to ask for JSON in the prompt. Respond only with a JSON object matching this shape. It works most of the time, which is the problem — most of the time is the failure profile of something you have to handle anyway.

You get a fenced code block one call in twenty. You get a leading "Here's the JSON you asked for:". You get a trailing comma, or a field named total_count where you asked for totalCount. Each is individually trivial and collectively means the caller is writing a tolerant parser, which is the job you were trying to avoid.

The deeper problem is where the error surfaces. A malformed response fails in the caller, after the work is done, when the only recovery is to throw the result away and re-run the whole step.

The actual mechanism#

Move the validation to the tool-call layer, so the shape is enforced before the result ever reaches you.

The mechanism is that the agent is not asked to write JSON — it is given a tool whose parameters are the schema, and required to call it. A response that does not fit cannot be emitted as a malformed answer, because it is not an answer at all; it is an invalid tool call, which the model sees and retries.

const verdict = await agent(prompt, { schema: VERDICT_SCHEMA })
//    ^ already an object. No parsing, no tolerant regex, no retry loop in the caller.

Two consequences matter more than the convenience:

The retry is free and local. The model corrects its own malformed call inside the same step, without the caller knowing there was a problem, and without re-running the expensive reasoning that produced the content.

The schema is a contract you can read. Anyone can see what the step returns by looking at the schema, rather than by reading the prompt and guessing what the prose usually contains.

The fix#

Three design choices that turned out to matter more than the schema itself.

Include the fields you will assert on, not the fields that describe the work. An early version of a verdict shape had summary, reasoning and confidence. None was checkable. The useful version had a boolean and an enum, and the prose moved to a field nobody branches on.

Let a shape be absent, and handle it. A dispatched agent can die on a terminal error or be skipped; the call then returns null rather than a shape. So every fan-out ends with a filter:

const results = (await Promise.all(tasks)).filter(Boolean)

Leaving that out produces a crash three stages later, in code that has no idea an upstream worker never ran.

Exclude fields deliberately. A schema is also a statement about what must not come back. The secret-handling audit here records the type of a credential and how it was entered, and has no field for its value — not because nobody would fill one in, but because a shape with no slot for a thing cannot carry it.

Enforced at the tool layer, the schema is a retry loop rather than a validation step — which is the difference between a contract and a hope:

type Schema = {
  required: string[];
  enums: Record<string, string[]>;
  forbidden: string[];          // fields that must NOT appear, at any depth
};

const VERDICT: Schema = {
  required: ["isReal", "category"],
  enums: { category: ["real-behavior", "stub-behavior", "partial", "unclear"] },
  forbidden: ["value", "secret", "token", "rawOutput"],
};

function validate(obj: unknown, s: Schema): string[] {
  const errs: string[] = [];
  if (typeof obj !== "object" || obj === null) return ["not an object"];
  const o = obj as Record<string, unknown>;

  for (const f of s.required) if (!(f in o)) errs.push(`missing required field "${f}"`);
  for (const [f, allowed] of Object.entries(s.enums)) {
    if (f in o && !allowed.includes(String(o[f]))) {
      errs.push(`"${f}" is "${o[f]}", not one of ${allowed.join(" | ")}`);
    }
  }
  // Walk the whole tree: a forbidden field nested two levels down still carries it.
  const walk = (v: unknown, path: string) => {
    if (typeof v !== "object" || v === null) return;
    for (const [k, child] of Object.entries(v as Record<string, unknown>)) {
      if (s.forbidden.includes(k)) errs.push(`forbidden field "${path}${k}" present`);
      walk(child, `${path}${k}.`);
    }
  };
  walk(o, "");
  return errs;
}

async function callTyped<T>(ask: () => Promise<unknown>, s: Schema, attempts = 3): Promise<T | null> {
  for (let i = 1; i <= attempts; i++) {
    const raw = await ask();
    const errs = validate(raw, s);
    if (errs.length === 0) return raw as T;
    // Retrying at the TOOL layer is what makes this a contract. A caller that
    // parses and hopes gets whatever came back on the first attempt.
    console.log(`attempt ${i}: ${errs.join("; ")}`);
  }
  return null;        // exhausted — the caller MUST handle absence
}

Three agents in one fan-out, two of them producing something off-contract first:

attempt 1: "category" is "probably-real", not one of real-behavior | stub-behavior | partial | unclear
attempt 1: forbidden field "evidence.rawOutput" present
attempt 2: missing required field "isReal"

const results = (await Promise.all(tasks)).filter(Boolean);
console.log(`${results.length}/3 returned a valid shape`);

2/3 returned a valid shape

The filter(Boolean) is the line that is easiest to leave out and the one that decides whether a missing shape is handled or discovered later. One agent exhausted its attempts and returned null; without the filter that null travels three stages downstream and fails inside code that has no idea an upstream worker never produced anything. And note the second log line — a nested rawOutput was refused at depth, which is the only reason the schema's exclusion is worth more than a convention.

How we proved it#

The structured return from a conversion step, and what it let me assert without trusting the step:

frames    48
moving    true
bytes     97226

Each of those is a claim I checked independently — I extracted the frames, hashed them, and counted the distinct ones. That cross-check is only possible because the fields exist. Against a sentence, there is nothing to disagree with.

And then the part that makes this post honest.

moving: true was wrong. The field is derived from compressed size against a single frame, so a GIF whose frames differ in some trivial respect satisfies it. I checked one and found the frame at the end byte-identical to the frame at the start; the visible composition never changed in the way the animation was supposed to show. The shape was perfectly valid. The value passed its type. The claim was false.

That is the boundary of the whole technique: a schema constrains the form of an answer and says nothing about its truth. Everything a structured return buys you is in the handling — no parsing, local retries, a readable contract, assertable fields. None of it is evidence.

If anything, a well-formed shape is more dangerous than a paragraph, because moving: true reads as a measurement. The prose version — "the animation appears to move" — would have invited the scepticism the JSON suppressed.

The field type that forces a lie#

One more design choice, and it is the one I get wrong most often: reaching for a boolean when the honest answer has three states.

A boolean says the thing either happened or did not. Plenty of real checks have a third outcome — I could not determine this — and a boolean has nowhere to put it. So it gets folded into false, which reads as "the check failed", or into true, which is worse.

The verification vocabulary here is deliberately an enum rather than a flag:

gate-pass | fixture-test | maestro-authored | maestro-executed | pending-device-verify

The last value is the interesting one. pending-device-verify means the test exists and could not be executed, because no physical device was attached. It is explicitly defined as non-passing — it blocks a release exactly as a failure would — but it is not a failure, and the distinction is what tells you whether to debug the code or plug in a phone.

Collapse that to a boolean and you must choose between two wrong answers. false sends someone hunting for a bug that may not exist. true ships untested work. Neither is recoverable later, because the information that would have separated them is gone the moment it was written.

The same shape appears in a judging verdict — real-behavior | stub-behavior | partial | unclear — where unclear exists so the judge has somewhere to put low confidence instead of rounding it to an opinion. A field with no representation for I don't know guarantees that uncertainty is recorded as certainty.

So when a schema field is a boolean, it is worth asking once whether the world really has two states in it, or whether you are about to make a caller act on a distinction you threw away.

What this generalizes to#

Give every step that returns to a caller a shape, and enforce that shape where the call happens rather than where the result lands — so a mismatch is the model's problem to retry, not yours to parse. Put the fields you will branch on in it, leave room for absence, and leave out the fields that should never travel.

Then hold the contents to the same standard you would hold a sentence. The schema guarantees you received a boolean. It has no opinion about whether the boolean is true.

---

Related#

• Ask a second agent to refute the first — Getting at correctness once you have the shape
• Where the tokens actually went — What re-parsing prose costs across a long session

Keep reading#

One post a week on shipping software with AI agents — what broke, what the fix was, and the evidence. No digests, no course pitch. Subscribe.