What to log when the actor is a model

A week after a run, the transcript is useless and the sidecar files are not.

The transcript has everything — every step, every reconsideration, the whole narrative. The sidecars have about twelve fields each. The sidecars are what let me answer questions.

The case this post is about#

One recorded verdict:

{"frames": 48, "moving": true, "bytes": 97226}

moving is derived — computed from the compressed size against a single frame — and it is wrong. The animation it describes has a last frame byte-identical to its first.

The reason that was diagnosable a week later is the other two fields. Because frames and bytes were also recorded, I could go back, extract the frames, hash them, and find that 41 of 43 were identical. Had the record contained only moving: true, there would have been nothing to check it against — and nothing to indicate it needed checking.

This post follows that one line, because a transcript records what was said, and what you need a week later is what was decided and from what.

The symptom#

An agent's run produces an enormous amount of text and almost no record. Those are different things, and the difference shows up exactly when you need it: something looks wrong, and the question is why did it do that.

The transcript answers a related but useless question — what it said while doing it. Reasoning is narrative, it is long, it varies in vocabulary between runs, and it cannot be queried. Ask "which runs used the fallback path last week" and there is nothing to ask it of.

The failure I hit most often is the undiagnosable success: a run that completed, produced output, and left no way to determine which of several paths it took. A month later the output is wrong, and the only honest answer to "how was this made" is a shrug.

Why the obvious fix failed#

The obvious fix is to log more. Verbose mode, every step, every decision.

It produces a second transcript with worse prose. The volume defeats the purpose — a log nobody reads because it is enormous is functionally the same as no log, and it has the additional property of looking like diligence.

The second attempt is to log at the end — a summary of what the run did. This is more readable and loses the thing that matters: a summary is written by the same process that did the work, after the fact, and it records what the process believes it did. Every failure where the belief was wrong is invisible in it, which is precisely the failure class you are logging for.

The third is to log decisions as sentences. Chose the procedural fallback because the network was unavailable. Readable, and unqueryable a week later when you want to count how often that happened.

The actual mechanism#

Write a per-artifact record, at the moment of production, in a shape you can query.

Here is one from this session, trimmed to the fields that earn their place:

engine:            google-stitch
backend:           stitch-mcp
account:           a4
stitch_project_id: 18393347132512223580
source_sha256:     0a98e96c882cf46...
generated_at:      2026-09-30T06:40:48Z
license_status:    conditional
human_signed_off:  false

Four properties make that useful where a narrative is not:

It names the path taken, not the reasoning. engine: google-stitch is checkable. "I used the image producer" is not, and the distinction matters because a fallback path produces a different value in that field without anyone writing a different sentence.

It carries the inputs. The prompt and a hash of the source are in the record, so the artifact can be reproduced or compared without reconstructing what the run was thinking.

It is per-artifact, not per-run. The unit of later investigation is a thing — this image, this post, this build — and a run-level log forces you to reconstruct which run produced which thing.

It has no slot for what must not travel. The credential audit here records the type of secret and how it was entered, and has no field for the value. Not because nobody would fill one in — because a shape with no slot cannot carry it.

The fix#

The rule that did the most work, and it took a real failure to learn: log the measurement, not the verdict.

A conversion step in this pipeline reports whether an animation moves:

{"frames": 48, "moving": true, "bytes": 97226}

moving is derived — computed from compressed size against a single frame. It is also, sometimes, wrong: I found an animation where it reported true while the frame at the end was byte-identical to the frame at the start.

The verdict was undiagnosable on its own. What made it diagnosable was that frames and bytes were also recorded, so I could go back, extract the frames, hash them, and see that 41 of 43 were identical. Had the record contained only moving: true, there would have been nothing to check it against.

So for every derived field, log the quantities it was derived from. The verdict is convenient; the measurement is what survives the verdict being wrong.

The shape that enforces it is a record type where the verdict cannot be written without its inputs:

type Derived<V, M extends Record<string, number | string>> = {
  value: V;
  derivedFrom: M;        // required — the verdict cannot exist without its measurements
  method: string;        // required — how, so a future reader can re-derive
};

function movingFrom(frames: number, bytes: number, firstFrameBytes: number): Derived<boolean, {
  frames: number; bytes: number; firstFrameBytes: number; bytesPerFrame: number;
}> {
  const bytesPerFrame = Math.round(bytes / frames);
  return {
    // The heuristic that was WRONG: compressed size implies motion. Kept, because
    // it is cheap and usually right — but never recorded on its own.
    value: bytes > firstFrameBytes * 1.2,
    derivedFrom: { frames, bytes, firstFrameBytes, bytesPerFrame },
    method: "total-bytes > first-frame-bytes * 1.2",
  };
}

// The check that the heuristic could not make, available only because the inputs were kept.
function reDerive(d: ReturnType<typeof movingFrom>, frameHashes: string[]): string {
  const distinct = new Set(frameHashes).size;
  if (d.value && distinct <= 2) {
    return `CONTRADICTED: reported moving=true, but ${frameHashes.length} frames hash to ` +
      `${distinct} distinct value(s). method was "${d.method}"`;
  }
  return `confirmed: moving=${d.value}, ${distinct}/${frameHashes.length} frames distinct`;
}

The animation that was wrong, logged and then re-derived a week later:

const rec = movingFrom(43, 97_226, 71_004);   // frames, total bytes, first-frame bytes
console.log(JSON.stringify(rec));
console.log(reDerive(rec, hashFrames("anim-warmth.gif")));

{"value":true,"derivedFrom":{"frames":43,"bytes":97226,"firstFrameBytes":71004,
 "bytesPerFrame":2261},"method":"total-bytes > first-frame-bytes * 1.2"}
CONTRADICTED: reported moving=true, but 43 frames hash to 2 distinct value(s).
 method was "total-bytes > first-frame-bytes * 1.2"

The record disagreeing with itself a week later is the whole point. 97226 > 71004 * 1.2 is true, so the heuristic fired correctly on its own terms — and 41 of 43 frames were byte-identical, which the compressed total cannot see because a repeated frame still costs a few hundred bytes. Had the log contained {"moving": true} there would have been nothing to contradict, and the method string is what told me which heuristic to stop trusting rather than merely that something was off.

How we proved it#

The test is whether a question asked later can be answered without guessing. Three that came up in this session, all answerable from records rather than memory:

Which account produced this image, and with what brief? — in the sidecar, including the full prompt and the remote project id.

Did this cover come from the producer or the fallback? — one field. It matters: a measurement on 2026-09-29 found 303 of 304 assets had come from the fallback path while the documentation called it the exception. Nobody noticed for weeks, because the only signal was a field nobody had thought to record until then.

Is this animation actually animated? — answerable only because the frame count was logged beside the verdict.

The honest limitation: this is all self-reported. Every field above is written by the process whose behaviour it describes, so a step that is confused about what it did records that confusion faithfully. moving: true was not a lie, it was a sincere derivation from an insufficient heuristic. Telemetry buys you what the actor believed, in a queryable form — which is worth a great deal and is not the same as ground truth. The only escape is a record written by something other than the actor, and most of the time there isn't one.

The field I keep wanting and cannot have#

The most useful field would be what would have had to be different for this to go the other way — the threshold, the missing input, the branch not taken.

It is the question every later investigation actually asks, and I cannot log it, because at the moment of the decision the alternative is rarely represented explicitly. The run took a path; the path not taken has no data structure.

The closest approximation I have found is recording the inputs to the branch rather than the branch itself. network_available: false beside engine: procedural lets a reader reconstruct the counterfactual. It only works where the branch condition is a value rather than a judgement, which covers less than half the interesting cases.

What this generalizes to#

Log per-artifact rather than per-run, at the moment of production, in a shape that can be queried rather than read. Record the path taken as a value, the inputs beside every derived verdict, and nothing at all in a field whose existence would let a secret travel.

Then accept what it is: a faithful record of what the actor believed. That is enough to answer most questions asked a week later, and it is exactly not enough for the one case where the actor was confidently wrong — so keep at least one check whose evidence comes from somewhere the actor never touched.