Auto-fix needs a floor, not an opinion

An automated repair is either allowed to apply its changes or it is not. The usual way this gets decided is a flag, set once, by somebody weighing how much they trust the mechanism in general.

The mechanism is not uniformly trustworthy, and neither are the repairs.

The symptom#

A single switch governing a range of operations that have nothing in common except the word "fix". Reformatting an import block and rewriting a data-access path are both repairs; a setting that permits both because it permits repairs has not expressed anything useful.

The second thing is where the switch ends up. Turned on, it eventually applies something alarming and gets turned off; turned off, the mechanism does nothing and gets removed. Neither state is stable, and the oscillation between them is most of what happens to auto-fix systems.

The third is that the useful distinction is not about the repair's category. People reach for a list of allowed operations — formatting yes, logic no — and the list is a poor proxy, because a formatting change across four hundred files and a logic change in one well-understood place have very different risk profiles in the opposite direction from the categories.

The fourth is that the information needed to decide is available at the moment of the repair and nowhere else. How certain the mechanism is about this particular fix, and how much this particular change touches, are both knowable right then — and a flag set weeks earlier cannot carry either.

The fifth property is social rather than technical, and it is why these systems die. The person who turns the flag off is usually the person who was surprised by a change, and their reasoning is sound — they have evidence the mechanism does something they did not expect. The flag gives them exactly one way to act on that evidence, and it is the maximal one.

Why the obvious fix failed#

Allow-list the safe operations. The common approach and it is a static answer to a dynamic question. The same operation is safe in one instance and not in another, and the list cannot see the difference.

Require review for everything. Correct and it removes the point. If every repair needs a person, the mechanism has saved the typing and none of the attention, which is the expensive part.

Let the agent decide whether it is confident. Asks for a self-assessment with no external reference, and self-assessed confidence is both unreliable and uniformly high. It also puts the decision inside the thing being governed.

Cap the number of changes. A real improvement and only half the picture. Ten changes that are individually certain and ten that are guesses are the same number, and the cap treats them identically.

Start with everything in proposal mode and promote what proves safe. The cautious path, and it decays in a specific way: the promotions happen for the classes somebody noticed were tedious, which is a measure of annoyance rather than of risk, and the list that results looks principled.

The actual mechanism#

Decide with two measured quantities, not one opinion.

confidence   how well-determined is this specific fix?
blast        how much does it touch, and how reversible is it?
floor        apply unattended only above a threshold on both

The multiplication is the point. A highly certain fix with a large blast radius still waits for a person; a small, contained change with moderate confidence can proceed. Neither axis alone produces sensible behaviour, which is why single-flag systems feel wrong whichever way they are set.

Confidence has to come from something observable rather than from an assertion. A fix derived from a deterministic rule — this pattern always becomes that one — is high confidence by construction. A fix produced by inference about intent is not, however convincing its explanation, and the distinction is in the derivation rather than in how sure anybody sounds.

That distinction is also stable over time, which matters for a threshold somebody has to trust. A rule-derived fix is rule-derived next month too, whereas a self-assessed confidence drifts with phrasing, context length and whatever else is in the prompt.

Blast radius is the easier of the two to compute and the one people skip, probably because it feels mechanical. Files touched, whether the change is confined to one module, whether it is trivially revertible, whether anything downstream regenerates from it — all countable before the change is applied, none requiring judgement.

The reversibility term is the one worth being careful about. A change to a file under version control is trivially revertible; a change that triggers a regeneration, writes to a store, or gets published is not, and treating those as the same because both are "one file" is how a small blast radius turns out to have been large.

The sub-case that matters is what happens below the floor. The answer is not "do nothing" — it is "propose". A repair that cannot be applied unattended is still a repair somebody can review in seconds, and routing it as a proposal preserves the work. Systems that treat the floor as a gate on acting rather than on applying throw away most of their own value.

The proposal route is also where the floor earns its keep as a learning mechanism. Proposals that are consistently accepted are evidence that the floor is too strict for that class, and proposals consistently rejected say the opposite — a feedback signal a binary flag cannot produce, because nothing below its threshold is ever seen.

The fix#

derive     confidence from how the fix was produced, not from self-report
count      blast radius from what the change touches and how reversible it is
apply      above the floor on both axes, unattended
propose    below it — same work, routed to a person instead of applied

Both axes are computed before the change is applied, and neither asks the repair how sure it is:

/** Confidence comes from the DERIVATION, which is stable, not from a self-report, which drifts. */
type Derivation =
  | { kind: "rule"; id: string }        // this pattern always becomes that one
  | { kind: "codegen"; generator: string } // regenerated from a declared source
  | { kind: "inferred"; rationale: string }; // an argument about intent, however convincing

const CONFIDENCE: Record<Derivation["kind"], number> = {
  rule: 1.0,        // deterministic: the same input yields the same fix next month
  codegen: 0.8,     // deterministic given a source that could itself be wrong
  inferred: 0.3,    // the explanation is not evidence about the explanation
};

interface Change { files: string[]; modules: Set<string>; regenerates: boolean; published: boolean }

/** Blast radius is countable. It is the axis people skip because it feels mechanical. */
function blast(c: Change): number {
  // Reversibility dominates. A published change and a one-file edit are not the same risk
  // because both touch "one thing" — this is the term that makes a small blast radius honest.
  if (c.published) return 1.0;
  if (c.regenerates) return 0.8;      // the diff is small; what it causes is not
  const spread = Math.min(1, c.files.length / 20) * 0.5 + (c.modules.size > 1 ? 0.3 : 0);
  return Math.min(0.7, spread);
}

type Decision = { act: "apply" | "propose"; why: string };

function decide(d: Derivation, c: Change, floor = { conf: 0.75, blast: 0.5 }): Decision {
  const conf = CONFIDENCE[d.kind];
  if (conf < floor.conf) {
    return { act: "propose", why: `confidence ${conf} (${d.kind}) below ${floor.conf}` };
  }
  const b = blast(c);
  if (b > floor.blast) {
    return { act: "propose", why: `blast ${b.toFixed(2)} above ${floor.blast}` };
  }
  return { act: "apply", why: `${d.kind}, blast ${b.toFixed(2)}` };
}

And the decisions it made across one drain, which is the behaviour a single flag cannot produce:

$ deno task auto-fix --explain

apply    rule(no-hardcoded-string)      1 file,  blast 0.03   → applied
apply    rule(import-order)             7 files, blast 0.18   → applied
propose  codegen(kmp-screen-gen)        2 files, blast 0.80   → regenerates; routed
propose  inferred(handler intent)       1 file,  blast 0.03   → confidence 0.3 below 0.75
propose  rule(version-bump)             1 file,  blast 1.00   → published; routed

applied 2 · proposed 3 · 0 discarded

The last two rows are the ones a flag gets wrong in opposite directions. A one-file rule-derived version bump is the most certain fix in the list and still waits, because publishing is not revertible; a one-file inferred change is the smallest possible diff and still waits, because the smallness says nothing about whether the inference was right. trust_auto_fix: true applies both. false applies neither, and discards the three proposals along with them.

The fourth line is what makes the floor acceptable to set conservatively. If below-floor means nothing happens, every threshold argument is a fight about lost value; if below-floor means a proposal appears, the threshold can start strict and relax on evidence.

The thresholds themselves are less important than having the two axes at all. I have never been confident about the specific numbers, and the system behaved sensibly across a fairly wide range of them — which suggests the value is in the shape of the decision rather than in its calibration.

That is a convenient property, because calibration is the part nobody has time for. A system that only works at the right thresholds needs somebody to find them; one that works across a range needs somebody to pick two plausible numbers and move on, which is a much likelier thing to happen.

How we proved it#

The oscillation stopped, which is the practical evidence. A single-flag system we had been turning on and off settled once the decision became two-dimensional, because the cases that had been causing the switch to be turned off were now below the floor and proposing rather than applying.

The derivation-based confidence is checkable, which surprised me. Classifying fixes by how they were produced — rule-derived, pattern-matched, inferred — is mechanical, and it predicts which fixes turn out to be wrong better than any self-assessment I have seen.

What I have not measured is the false-negative cost: how many correct fixes sat as proposals waiting for somebody. That is the floor's main cost and I do not have a number for it, which means I cannot say whether my thresholds are badly placed.

The two axes were not equally hard to build. Blast radius took an afternoon and has needed almost no maintenance; confidence took several attempts, and the versions that failed were all variations on asking the mechanism how sure it was.

What I have not solved: blast radius for changes whose consequences are not local. A one-line change to a shared constant touches one file and affects everything that reads it, and my blast calculation counts files. I have handled this with manual exclusions, which is exactly the static list this post argues against.

And a limitation worth naming: both axes measure properties of the change, not of its correctness. A confident, contained fix can still be wrong — the floor governs how much damage a wrong fix can do unattended, which is a different and more modest claim than that the fixes are right.

The thing that reframed it was noticing that the question "should this apply automatically" had been treated as a question about trust, and trust is not a quantity anybody can act on. Replacing it with two things that can be counted turned an argument into a calculation, and arguments about thresholds are much easier to have than arguments about trust.

It also made the system's behaviour explicable. "This did not apply because it touches eleven files" is a sentence somebody can agree or disagree with; "the auto-fixer decided not to" is not, and a mechanism whose decisions cannot be discussed accumulates resentment until it is switched off.

What this generalizes to#

A permission that is a single switch over a range of actions will be set wrong for most of them.

The same structure applies to anything granting autonomy: deployment approvals, data retention rules, spending limits, access control. In each case the binary is a proxy for a calculation nobody wrote down, and the calculation is usually some version of how certain we are against how much it costs if we are wrong.

So the question to ask of any automation permission is what two quantities would actually decide this. If the answer is available at the moment of the action and the switch was set weeks earlier, the switch is throwing away information that was right there.