Agents

Say sorry once and it happens again

When an agent is told it behaved wrongly, an apology changes nothing. The response that holds is an edit to the runtime plus a gate that catches a recurrence.

MobileByteSensei 10 min read
A stone stair with one deeply worn tread freshly replaced by a new cut stone, photographed head-on at tread level, and a folding warning stand collapsed and set aside against the wall — the masonry clean and the new tread flush
agentsguardrailsprocess

Somebody points out that a command stopped halfway through and asked a question it should not have asked. The response is an acknowledgement, an explanation of what went wrong, and an assurance that it will be handled differently.

Nothing in the system has changed, and the next run behaves identically.

The symptom

The exchange feels resolved. The problem was named, understood, accepted — and the artifact that produced the behaviour is untouched, so the behaviour is not a past event but a standing property.

The second thing is that the apology is a very good imitation of a fix. It demonstrates understanding, it is specific, it usually includes a correct account of the cause. Everything a fix would contain is present except the fix.

The third is the recurrence pattern, which is what makes this expensive. The same behaviour appears again, gets flagged again, is acknowledged again — and each cycle costs somebody's attention and produces a slightly stronger impression that the thing is unreliable in a general way, rather than that one specific instruction is missing from one specific file.

The fourth is the escalation that does not happen. Because every instance is handled as a fresh incident, nothing accumulates. There is no record that this is the third time, no mechanism that treats a repeat differently, and therefore no pressure to do something structural.

By the fourth occurrence the cost has changed character. It is no longer the behaviour that is expensive; it is that somebody now has to watch for it, and watching is a permanent tax that the original defect did not impose.

The watching also has a second-order effect that is worth naming. Once somebody is checking for a known behaviour, they stop reading the rest of the output as carefully — attention is finite, and it has been spent in advance on a specific thing. The defect costs more than its own occurrences.

Why the obvious fix failed

Remember to do it differently. The thing everybody means, and it is not a change to anything. In a system where each run starts fresh, memory is whatever is written in the files that run reads, and an intention expressed in a conversation is not one of them.

Add it to a general instruction document. Better, because it persists — and it dilutes. A document that accumulates one line per past incident becomes long, is loaded less carefully, and its individual instructions carry progressively less weight.

Make the instruction more emphatic. Capitals, "NEVER", "HARD RULE". Sometimes works, and when the same behaviour recurs after an emphatic instruction, the lesson is that emphasis was not the missing ingredient and more of it will not help.

Handle it in review. Puts a person in the loop for a behaviour that is, by construction, going to recur. It converts a defect into a chore and spreads the cost over everybody who reviews.

Write a test. The right shape and the wrong target, when the behaviour belongs to a command's runtime rather than to code. A test suite that exercises functions has nowhere to put "this command should not have paused here", so the instinct gets abandoned rather than redirected.

The actual mechanism

Treat a flagged behaviour as a defect in a runtime artifact, and respond with the two things a defect gets: a change at the root, and a check that fails if it comes back.

edit    change the command's own runtime, at the cause, not the symptom
gate    an evaluator that fails when the behaviour reappears
canary  a fixture that reproduces it — red before the edit, green after

The second line is what distinguishes this from a careful instruction. An instruction is a request; a gate is a mechanism. If the behaviour recurs, something fails — which means the third occurrence cannot happen quietly, and the question stops depending on whether anybody is watching.

The third line is what makes the gate trustworthy, and it is the step most often skipped. A gate written after the fix, which has never been seen to fail, might be checking nothing at all; a gate proven against a fixture that reproduces the original behaviour is known to detect it.

The sub-case that matters most is the second flag of the same class. If a behaviour is reported again after it has supposedly been healed, that is information: the previous fix was at the symptom. The correct response is not to apply a similar fix again but to escalate — a stronger guard, a different enforcement point, a mechanical block rather than an instruction.

There is a question underneath all of this about where the root actually is, and getting it wrong produces fixes that look structural and are not. A command that asks an unwanted question might be missing a rule, or might be loading one that permits the question, or might be reaching the point at all because an earlier step did not do its job. Editing the first of those is a symptom fix dressed as a root fix.

The only reliable way I know to tell them apart is to ask what would have to be true for the behaviour to be impossible rather than merely discouraged. If the answer is "the instruction is followed", the fix is at the symptom however structural the diff looks; if the answer is "this branch cannot be reached", it is at the root.

The fix

root      locate the artifact whose content produced the behaviour
edit      change it there — the file that will be read next run
lock      add an evaluator gate plus a red/green canary for the class
escalate  a repeat of the same class means the last fix was a symptom fix

The fourth line only works if repeats are countable, which means the heal has to be recorded as data rather than as a commit message:

interface Heal {
  class: string;          // the DEFECT class, not this instance of it
  artifact: string;       // the file whose content produced the behaviour
  gate: string | null;    // the evaluator that now refuses it
  canary: string | null;  // the red/green fixture pair that proves the gate works
  at: string;
}

/**
 * Called when somebody reports the behaviour again. The return value is the interesting part:
 * a second report of the same class is evidence about the FIRST fix, not just a new bug.
 */
function onReport(cls: string, log: Heal[]): { verdict: string; next: string } {
  const prior = log.filter((h) => h.class === cls);

  if (prior.length === 0) {
    return { verdict: "first report", next: `edit the artifact, then add a gate and a canary` };
  }

  // No gate means the previous fix was prose — an edit that asked the next run to behave,
  // with nothing refusing it. This is the common case and it is not a repeat offence.
  const ungated = prior.filter((h) => h.gate === null || h.canary === null);
  if (ungated.length > 0) {
    return {
      verdict: `${prior.length} prior heal(s), ${ungated.length} without a gate or canary`,
      next: `the prior fix was unenforced — add the gate for ${ungated[0].artifact}`,
    };
  }

  // Gated and it happened anyway: the gate does not cover this variant, so the gate is the
  // thing to change. ESCALATE means strengthen the guard, not apologise harder.
  return {
    verdict: `${prior.length} prior heal(s), all gated — the guard is too narrow`,
    next: `extend ${prior.at(-1)!.gate} and add a canary for THIS variant before fixing it`,
  };
}

And the three outcomes, from the log of one command's defect classes:

$ deno task heal --report forbidden-halt-mid-drive

3 prior heal(s), all gated — the guard is too narrow
next: extend framework-verify-user-heal.sh and add a canary for THIS variant before fixing it

  2026-07-18  runtime edited, gate added, canary added
  2026-08-13  runtime edited, gate added, canary added
  2026-09-01  runtime edited, gate added, canary added

Three gated heals of one class is a different finding from three apologies, and it points somewhere specific: the class is real, the fixes were enforced, and the enforcement keeps missing a variant — so the work is in the predicate, not in the runtime. Without the log this reads as "this keeps happening", which is the sentence that produces a fourth apology.

The fourth line is the one worth building a habit around, because it is the only part that self-corrects. Any individual diagnosis can be wrong, and the recurrence is the evidence — treating a repeat as "the same problem again" rather than "my fix was in the wrong place" is how a defect survives three rounds of sincere attention.

What this costs is that every flagged behaviour becomes a small piece of engineering rather than a conversation. That is a real cost and it is almost always smaller than the alternative, because the alternative is paid repeatedly and by somebody whose attention is the scarcest thing in the system.

The part that is easy to get wrong is scope. A gate written to catch the exact reported instance catches the exact reported instance; the useful version catches the class, which requires deciding what the class is — and that decision is where most of the thinking goes.

The pull is toward over-narrow, because the reported instance is concrete and the class is not. A useful discipline is to write down two other behaviours that would be the same mistake, and check the gate catches those too — if it does not, the class has been drawn around the example rather than around the defect.

How we proved it

The recurrence is the measurement, and it is unambiguous. A behaviour that was flagged, fixed and never seen again is a fix; one that was flagged, acknowledged and seen again is not, and the distinction needs no instrumentation beyond noticing.

The canary proves the gate, and it has caught gates that did not work. Writing the fixture first and watching it fail is a few minutes, and more than once that step revealed a gate that would have passed on the original defect — which would have been worse than no gate, because it would have been believed.

What I have not measured is how many flagged behaviours never recur anyway. Some proportion of them are genuinely one-off, and for those the whole apparatus is overhead; I do not know the ratio and I apply the same response to all of them, which is certainly over-engineering some of the time.

What I have not solved: the behaviour that cannot be expressed as a gate. Some flags are about judgement — proportion, tone, when to stop — and I have not found a way to write a mechanical check for most of them. Those get an instruction and an honest acknowledgement that it is weaker.

And a limitation worth naming: this treats the agent's behaviour as a function of its runtime artifacts, which is mostly true and not entirely. Some behaviour comes from the model rather than from any file, and for that class the runtime edit is a constraint applied from outside rather than a correction, and it will be approximate.

The thing that reframed it was noticing how much better the apology feels to everybody involved than the engineering does. It is immediate, it demonstrates understanding, and it closes the interaction cleanly — all of which are properties of a good conversation and none of which are properties of a fix.

Which is probably why it is the default. The failure mode is not carelessness; it is that the socially correct response and the technically correct response are different, and the socially correct one arrives first and feels complete.

Noticing that is most of the remedy. The apology is not wrong and does not need to be suppressed; it needs to be understood as the opening of the response rather than the whole of it, which is a much easier thing to hold onto than a prohibition against acknowledging anything.

What this generalizes to

A correction that does not change an artifact changes nothing that will run again.

The same gap appears whenever a process depends on people remembering an incident: the post-mortem with action items nobody owns, the style decision made in a review thread and never written into a linter, the operational lesson that lives in one person's head until they leave. In each case the understanding is real and the mechanism is unchanged.

So the question to ask after any correction is what file is different now. If the answer is none, the correction was a conversation — valuable for understanding, and the next run has not heard it.

One post a week, no digest

New writing when it lands. No roundups, no course pitch, no "10 tips".

Follow by RSS