What should an AI agent never be allowed to do?

I have never successfully predicted which capability would cause damage.

Every restriction in the permissions model I actually use was added after something went wrong, and the list looks nothing like the one I would have written in advance.

TL;DR#

• Some actions should be unavailable rather than discouraged — deletion, publication, anything irreversible.
• A permission model is the difference between an agent that makes a mistake and one that makes an unrecoverable mistake.
• Where to draw the line, and why 'ask first' is not the same as 'cannot'.

The case this post is about#

One command, run to print the key names in a properties file:

cut -d= -f1 upload_keystore.properties

The file was malformed — one raw value, no delimiter — so cut returned the field unchanged and a signing secret went into the transcript.

Nothing blocked it. The rule in force at the time named three commands nobody should run on a credential file: cat, head, tail. cut was not among them, because the list was written by somebody imagining how a secret gets read rather than enumerating the ways it can.

That is the case this post follows, and it is representative: every restriction in the permissions model I actually use was added after an incident, and none of them came from a threat model. The interesting question is what that says about how to write the next one.

The symptom#

The natural way to design permissions is to imagine what could go wrong and restrict accordingly. It produces a list that is simultaneously too broad and too narrow.

Too broad, because imagined risk is unbounded — anything can be argued into the dangerous column, and a model that blocks a third of ordinary work gets bypassed as a matter of routine. Too narrow, because the imagination reaches for dramatic failures (delete the database, exfiltrate the keys) and real incidents are mundane.

Here is what actually cost me something, in order of expense:

• an improvised branch switch that discarded roughly three hours of untracked work
• a build daemon killed because a build looked stuck, costing a 15–30 minute cold rebuild
• a command run to list key names in a properties file, which printed a signing secret
• a keychain entry deleted during cleanup, which broke code signing until it was repaired

Not one of those is a capability I would have flagged. Three of them are cleanup.

Why the obvious fix failed#

The obvious fix is a broad denial: no writes outside a sandbox, no shell, confirm everything.

It fails on usefulness in a way that is easy to underestimate. An agent that must ask before every file write is not safer, it is a slower way to do the work yourself, and the operator learns to approve without reading — which is the state you least want them in when something genuinely dangerous appears.

The second attempt is severity tiers: low-risk auto, medium confirm, high refuse. The classification is the problem. git checkout -b is a low-risk command by any reasonable reading, and it is the one that cost three hours. Severity describes the operation; damage comes from the operation's interaction with state the classifier cannot see.

The third is to trust the agent's judgement, with the reasoning that a capable model will not do anything reckless. It did not do anything reckless. It ran a branch command, killed a stuck daemon, and listed field names — each a defensible local decision whose consequence lived elsewhere.

The actual mechanism#

Three tiers, and the third is the one most permission models are missing.

Permitted — the default and by far the largest surface. Read anything. Write source. Run tests, builds, generators. Reversible things that git can undo.

Gated — allowed to happen, but only through a sanctioned path that carries the guard. Git mutations go through session scripts rather than raw commands. Secret values move through a designated intake rather than a shell pipe. The capability is not removed; the improvised route to it is.

Unreachable — removed from the model's reach entirely, not gated. A set of commands here are marked so they cannot be invoked by the model at all: publishing a release, pushing a secret, merging a pull request. They are not in the list of things it can decide to do, so no amount of reasoning arrives at them.

That third tier matters because gating still puts the decision in the agent's path — it can consider the action, plan around it, and route toward it. Unreachable means the question never comes up.

The fix#

Two properties do more work than the tiers themselves.

Every bypass requires a human acting in their own shell. Not a flag the agent can pass, not an environment variable it can set inline — an export typed by a person, or a short-lived consent latch granted after an explicit confirmation and expiring on its own. An inline forgery is detected and refused. The rule is simple: if the thing being restricted can grant itself the exception, there is no restriction.

The latch is the interesting half, because a permanent human-only export is too coarse for work that legitimately needs one guarded operation:

LATCH="$STATE/consent-$OP.json"   # <!-- preflight-ignore --> a path, not a secret
TTL=1800        # 30 minutes. Long enough for the operation, short enough not to linger.

grant_consent() {                   # reachable ONLY from the confirmed interactive path
  printf '{"op":"%s","granted_at":%s,"by":"%s","pid":%s}\n' \
    "$1" "$(epoch)" "${SUDO_USER:-$USER}" "$PPID" > "$LATCH"
  chmod 600 "$LATCH"
}

consent_valid() {
  [ -f "$LATCH" ] || return 1
  age=$(( $(epoch) - $(json_get "$LATCH" granted_at) ))
  [ "$age" -lt "$TTL" ] || { rm -f "$LATCH"; return 1; }   # self-expiring, never self-renewing
  [ "$(json_get "$LATCH" op)" = "$1" ]                      # scoped to ONE operation
}

The forgery the model is designed around is an agent writing the latch itself, which fails because the granting path is only reachable from a confirmed human prompt:

$ agent attempts guarded-op
refusing: no consent latch for 'training-fold'
  a human confirms this once; the latch lasts 30 minutes and expires on its own

$ (human confirms) ; agent attempts guarded-op
proceeding under latch granted 00:04 ago by therajanmaurya

The expiry does real work. A permanent grant drifts into a permanent permission, and the only thing anybody remembers about it later is that it was once justified.

The tiering above the latch matters less than it looks, and I would resist elaborating it. Three levels — unrestricted, guarded-with-latch, human-only — have covered everything so far, and each additional level is a judgement somebody has to make correctly at the moment they add a new operation. A model with seven tiers is a model where things get filed by vibe.

What decides the tier is not how dangerous the operation feels but whether its effect is recoverable and whether it is outward-facing. Deleting a local build directory is destructive and costs minutes; publishing to a live audience is a single call and cannot be taken back. Those two intuitions pull in opposite directions from "how scary is this", and the second one is the one that should win.

The audit trail is the part I would not skip even though it has never once been read in anger. Every latch grant records who, when, and for what, which costs nothing and means that the question "who allowed this" has an answer. Systems where that question has no answer tend to acquire permissions that nobody can account for, and the acquisition is always individually reasonable.

The honest gap is that none of this constrains what happens within an allowed operation. A latch for one guarded write does not stop that write being wrong, and the model has nothing to say about it. It governs whether the door opens, not what walks through — which is a narrower claim than a permissions model usually implies, and the right one to make. no restriction.*

On the genuinely dangerous surface, the list is an allowlist. This one I learned the hard way. The secrets rule began as never cat, head or tail a secret file — three verbs. Then a command ran to print key names:

cut -d= -f1 upload_keystore.properties

The file was malformed, cut returned the whole line, and a signing secret landed in the transcript. cut is not cat. Neither is awk, sed, grep, od, xxd, strings, rev, tr, paste, base64, openssl, jq, dd, or tee, and the list of ways bytes leave a file is not one anyone finishes.

So on that surface the model inverted: a command touching a secret path is refused unless it is one of a small set of metadata-only operations — wc -c, stat, test -f, ls, shasum — or a sanctioned script. The insight that makes it liveable is that metadata always suffices to verify materialization. If a check seems to need the value, the check is wrong.

How we proved it#

The direct test is whether the restrictions fire in ordinary work rather than in contrived cases. Two fired on me while I was writing this batch of articles — one blocking a shell tool against the specification layer, one refusing a documentation file three lines over its cap. Both refusals improved the result.

The more interesting evidence is the shape of the rules, because it is not the shape I would have designed:

Every row is a receipt. That is the property I would keep if I could keep only one — a rule with a cost attached is a rule nobody argues with, and a rule nobody can attach a cost to is usually imagination.

The honest limitation is structural and unavoidable: an incident-derived model is always exactly one incident behind. The first occurrence of any new class is unprotected by construction, because the mechanism for adding protection is the occurrence. I have no answer to that. What I have is a cheap path from incident to rule — the heal is expected to land the same day, with a check that fails if the fix is absent — so the cost is bounded at one occurrence rather than a recurring tax.

The rule I got wrong in the other direction#

For balance, a restriction that was too aggressive.

The guard protecting the specification layer matches on the command string, so it refused an edit to a temporary file merely because the replacement text mentioned a protected path. Nothing in that command could have touched anything real. It was, correctly, a refusal of something harmless.

I narrowed it — behind nine cases written first, each reproducing a shape the guard must never miss — and the narrowing is the first one that file has ever had, against seven prior widenings.

The asymmetry there is worth stating plainly, because it explains why these models drift toward strictness: a false block costs one extra command and announces itself immediately. A false pass costs a mangled source of truth and announces nothing. Given that ratio, over-blocking is the correct default — right up until the false positives get frequent enough that people start routing around the guard, at which point you have traded a visible cost for an invisible one.

What this generalizes to#

Write the permissions model from your incident log, not from a threat model, and attach the cost to every rule so the list stays defensible. Keep the permitted surface wide, gate by removing the improvised route rather than the capability, and put the genuinely irreversible things somewhere the agent cannot reach at all.

Then check who holds each bypass. If the restricted party can grant its own exception, you have written documentation. And when a restriction starts refusing honest work, treat that as a real cost too — the model that gets routed around protects nothing.

---

Related#

• Guidance is not enforcement — hooks are — Making the boundary something the model cannot talk its way past
• Every agent action should be safe to run twice — For the actions that stay available

Keep reading#

One post a week on shipping software with AI agents — what broke, what the fix was, and the evidence. No digests, no course pitch. Subscribe.