Running an AI agent in CI without it spending your budget

The defining property of an agent in CI is not the sandbox or the credentials.

It is that there is nobody to ask. Every place the interactive version would surface a question has to become something else, and the interesting failures are the places where it quietly became neither an answer nor a failure.

TL;DR#

• An agent in CI has no human watching it, so cost and blast radius both need hard ceilings rather than good intentions.
• What to bound: tokens, wall-clock, tool surface, and what it may write.
• The failure modes that only appear when nobody is in the loop.

The case this post is about#

One credential, required by one verification step, in a run with nobody to ask:

$ ci verify
error: MEDIUM_SESSION is not available.
  looked in: env, vault
  this run cannot proceed without it — a default here would produce a run
  that completes and verifies nothing.

Interactively this is a prompt. In CI there is no prompt, so the step has exactly two honest options: resolve the credential from somewhere declared, or fail with the reason.

The third option is the one that ships by accident. A step that cannot find its credential and continues anyway — skipping the check, logging a warning, exiting 0 — produces a green run that verified nothing, and the green is indistinguishable from a run that verified everything.

The symptom#

Interactive runs have a pressure valve. Something is ambiguous, the agent asks, a human resolves it, the run continues. That valve is invisible until you remove it, and then every gate that depended on it has to do something new.

There are exactly three things a gate can do without a human, and only two of them are acceptable:

• resolve autonomously — the gate has enough information to decide
• fail hard — stop, with a named reason, having changed nothing
• degrade — proceed with less than it needed, silently

The third is the one that hurts, because it produces a green run. A step that could not fetch its configuration and continued with defaults, a check that could not reach the device and reported nothing rather than failure, a fetch that timed out and used a cached value from last week.

Why the obvious fix failed#

The obvious fix is to make the agent more autonomous — give it enough permission and context to resolve anything it meets.

That is the wrong direction, and expensively so. Every capability added to close a question also widens what an unsupervised run can do wrong, and CI is precisely where you have the least observation. The interactive session has a human watching the output; the CI run has a log nobody opens unless it goes red.

The second attempt is a global timeout. It bounds the damage in wall-clock and not in kind — a run that spends forty minutes doing the wrong thing confidently is stopped at forty minutes having done the wrong thing.

The actual mechanism#

Three controls, in the order they earn their place.

A budget that throws. Not a warning, not a log line — a hard ceiling where further work raises rather than continuing:

while (budget.total && budget.remaining() > 50_000) { ... }

The budget.total && is the part I got wrong first. With no budget configured, remaining() is infinite and that loop runs until it hits a completely different backstop — in this framework a lifetime cap of 1000 dispatched workers, set far above any real workflow precisely so that reaching it means something has gone wrong rather than something is large.

Reproduce locally before pushing. The loop worth killing is push → wait for CI → read failure → fix → push, at roughly 45 minutes per cycle. The alternative is to run the same workflows locally in a Linux container and iterate in seconds. A hook here refuses a push while a CI fix is in flight and not yet green locally, which makes the slow loop mechanically unavailable rather than discouraged.

Refuse to fake what cannot run. Local containers cannot run macOS, iOS or Xcode jobs. The sanctioned behaviour is to skip them and say so loudly — never to report them green because they did not fail.

The fix#

The principle underneath all three: an unavailable input is a failure, not a default.

That sounds obvious written down and is violated constantly, because the degraded path is almost always the more helpful-looking one at the moment it is written. A missing config with a sensible default lets the run continue. That is the entire problem.

Expressed as a resolver, where the absent case has no fallback branch to fall into:

type Source = "env" | "vault" | "config";

export function require(name: string, sources: Source[]): string {
  const tried: string[] = [];
  for (const s of sources) {
    const v = read(s, name);
    if (v) return v;
    tried.push(s);
  }
  // NO default. NO empty string. NO "" that a downstream check reads as present.
  throw new Error(
    `${name} is not available.\n` +
    `  looked in: ${tried.join(", ")}\n` +
    `  this run cannot proceed without it — a default here would produce a run\n` +
    `  that completes and verifies nothing.`,
  );
}

// The degraded path, when there IS a legitimate one, is NAMED in the result.
export function requireOrDegrade(name: string, sources: Source[], degraded: string) {
  try { return { value: require(name, sources), mode: "full" as const }; }
  catch { return { value: degraded, mode: "DEGRADED" as const, missing: name }; }
}

The difference on a CI runner with no browser session configured:

$ ci verify
error: MEDIUM_SESSION is not available.
  looked in: env, vault
  this run cannot proceed without it — a default here would produce a run
  that completes and verifies nothing.

$ ci verify --allow-degraded
mode: DEGRADED (missing MEDIUM_SESSION) · remote list is a LOWER BOUND

The second form is acceptable precisely because the degradation is in the output. The version this replaces returned the same lower bound with mode: full, and every count downstream was quietly about a smaller world.

The reason the degraded-by-default version is so attractive is worth naming, because it is not laziness. Writing the fallback is the helpful thing to do at the moment you write it: the run continues, nobody is blocked, and the alternative — a hard failure on a machine somebody else configured — feels like an unnecessary obstacle you are imposing on a colleague. Every instinct points toward the default.

What that instinct misses is who pays and when. The fallback is written once, by somebody who knows exactly what it means. It is then read thousands of times by runs that have no idea a fallback occurred, and by people reading those runs' output as though it described the full thing. The cost is deferred and distributed, which is precisely the shape of cost that never gets attributed back to the decision that caused it.

There is a narrower version of the rule that I find easier to apply in the moment: a default is acceptable when it cannot change the meaning of the result, and unacceptable when it can. A default timeout is fine — the run means the same thing either way. A default credential, a default empty list, a default "assume complete" — each of those changes what the output is about, and that is the line.

The CI case sharpens it because nobody is watching. On a developer machine a degraded run gets noticed, eventually, because the person running it has context. In CI the output is read by a gate, and a gate reading mode: full on a degraded run is the entire failure in one line.

The implementation detail that makes this stick is that mode is part of the return type rather than a log line. A log can be ignored by the next step; a field has to be destructured, which means somebody writing the consuming code has to decide what to do with it. That is a small amount of friction placed exactly where the decision belongs.

The test I use now: for every input a step needs, ask what happens if it is absent. If the answer is "it proceeds" and the output does not carry a field saying so, that is a degradation, and it will be discovered by someone downstream wondering why a value is wrong.

How we proved it#

The best evidence arrived unplanned, in the middle of writing this batch.

An image producer needs an API key from an encrypted vault. The vault's session had expired, so the auth gate tried to prompt for a master password, found no interactive terminal, and aborted:

⚠ TTY missing — auto-promoting to a dialog
BW_SESSION empty after dialog — aborting
StitchKeyUnavailable: vault resolve failed

That is the behaviour you want, and it is worth being precise about why. The gate had three options and took the right one. It did not proceed without the key. It did not fall back to an unauthenticated path. It did not write a placeholder asset and mark the work done. It stopped, named the missing thing, and printed the command that fixes it.

The cost was real — three finished articles are sitting unillustrated as I write this. The alternative was three articles shipped with broken images and a green run, discovered by a reader.

This is also the concrete version of a caveat worth internalising: interactively-authenticated services are frequently absent in headless runs. Anything that works because you once approved a dialog is a dependency that does not exist in CI, and the time to find out is not at 3am.

The honest limitation: my run was not actually headless. It was an interactive session where one subsystem happened to have no terminal, which is the easy version — a real CI run would have failed the same way but with nobody reading the message for hours. Everything above about failing loudly assumes someone eventually reads the log, and a hard failure nobody looks at is just a slower silent one.

The control I have not needed yet#

For completeness: the sandbox, which is where most discussions of this topic start.

I have not found it to be the interesting control, and I think the reason generalises. A sandbox bounds where damage lands — filesystem, network, credentials — which matters enormously for untrusted code and much less for a run whose whole purpose is to modify your repository and push. The agent is inside the blast radius by design.

What has actually prevented damage is narrower: the specific commands that cannot be improvised, the budget that throws, and the refusal to proceed on a missing input. Those are about what it may do rather than where, and in a system where the agent is meant to be changing real things, the what is the axis that carries the risk.

What this generalizes to#

Before running an agent unattended, walk its gates and ask what each one does with no human present. Anything that can resolve itself, let it. Anything that cannot must fail hard with a named reason — and the failure must name the fix, because the person reading it will be reading it cold.

Then give it a budget that raises rather than warns, and make the slow feedback loop mechanically unavailable rather than merely discouraged. The runs that hurt are not the ones that stop; they are the ones that continue with less than they needed and report success.

---

Related#

• Where the tokens actually went — Where the spend actually goes when you measure it
• What should an agent never be allowed to do? — Bounding what it can reach, not just what it can spend

Keep reading#

One post a week on shipping software with AI agents — what broke, what the fix was, and the evidence. No digests, no course pitch. Subscribe.