What your coding agent costs is an architecture problem, not a billing one

TL;DR#

Shortening prompts is the lever everyone reaches for, and it is close to the smallest one available.

The costs that matter are structural: which engine a task is routed to, how much context each worker is handed, and whether that context is paid once or once per worker. Measured in one codebase, those three decided roughly two orders of magnitude more than wording ever did.

The clearest case: a routine commit that cost 180,000 tokens and seven and a half minutes, because it was routed to a model rather than to the script that already did it.

The case this post is about#

A three-repository commit. Source, then workspace, then framework, bottom-up, with pull-request discovery and a branch-naming convention. Entirely mechanical, done dozens of times a week.

There were two ways to perform it. A deterministic bash script owned the whole procedure:

bash core/scripts/git-session-commit-lite.sh --all

Pure bash. No model call. Seconds.

Or the same work could be routed to a language model, which would read the command's runtime documentation, reason through its phases, and carry them out. That is the right path for a genuinely hard case — a rebase conflict, a repository in an unexpected state.

It was used for the routine case. It cost ~180,000 tokens and 7½ minutes, and produced exactly what the script produces in seconds.

Nothing was wrong with the prompt. The prompt was excellent. The routing was wrong, and no amount of prompt work would have recovered a hundredth of what the routing decision cost.

The symptom#

Cost problems with agents do not announce themselves, because every individual run looks reasonable. The bill is monthly and the runs are hourly, so by the time the number is visible the decisions that produced it are weeks of context behind you.

What you see instead is a set of plausible observations with no obvious common cause:

• Runs take minutes where the underlying work takes seconds
• The same operation costs wildly different amounts on different days
• Parallelising a job makes it more expensive per unit of work, not less
• A prompt-shortening pass saves 4% and nobody can tell afterwards

That last one is the tell. If careful prompt work moves the number by single-digit percentages, the number is not being set by the prompt.

Why the obvious fix failed#

"Shorten the prompts." Real, and tiny. Tightening a generator's instructions saved a few hundred tokens per call against a context pack of nearly 464,000 bytes. It is optimising the label on a container.

"Use a cheaper model everywhere." This confuses unit price with cost. The expensive run above was expensive because it did 180,000 tokens of reasoning about a procedure that needed none — moving that to a cheaper model makes the same wrong routing slightly less painful, and quietly degrades every task that genuinely needed the capability.

"Parallelise it." Fan-out multiplies the per-worker context cost. Five cold workers each loading the same background pay for it five times; one warm worker loads it once. Parallelism buys wall-clock and is often worth it, but it is a cost increase dressed as an optimisation, and treating it as a saving is how a fan-out gets chosen for a job that was never slow.

"Cache more." Helps, and only inside a context that is already correctly scoped. Caching a pack that should never have been assembled is faster waste.

Each of these operates on the content of a call. The decisions that set the bill happen before any call is made.

The actual mechanism#

Three structural costs, in descending order of what they decided here.

1 — Routing: which engine runs this at all. The largest factor by a wide margin, and the cheapest to get right. Deterministic procedures belong in deterministic code. A model is for the cases a script cannot enumerate: ambiguity, recovery, judgement. The 180,000-token commit was a routing error, and routing errors are invisible because the output is correct — you pay a thousand times over for a right answer.

The rule that came out of it is blunt: if a sanctioned script exists for the task, run the script. Reserve the reasoning path for halts, conflicts, and first-time setup.

2 — Context scope: what each worker is handed. A shared context pack was built once and given to every worker regardless of what it was doing. Scoping it by task type cut it from 464,099 to 405,506 bytes — about 14,600 tokens per worker, 12.6%.

That number is worth sitting with, because the estimate before measuring was 40%. The pack carried files that looked obviously irrelevant to most workers, and removing them turned out to be worth under a third of what intuition said. Measured, not estimated, and the gap between the two is the whole argument for measuring.

The inverse failure is the dangerous one. A pack trimmed too far makes a generator improvise with nothing announcing that it is improvising — so the saving is real and the cost lands later, in output nobody can trace back to a missing file. Trimming context without a mechanism that declares what was omitted trades a measurable cost for an unmeasurable one.

3 — Warm versus cold. The same work split across five cold workers pays the setup cost five times. One warm context pays it once and keeps everything already loaded. Cold fan-out is sometimes right — genuine isolation, genuine parallelism — but "run it in parallel" is a latency decision with a cost attached, and it is routinely described as though it were free.

And the one with no output at all. Today a generation service accepted 16 calls across 25 minutes and returned nothing on every one. Each was billed-shaped work with zero product. The design question that exposes is retry policy: a loop with no ceiling converts an outage into spend. The guard is a terminating condition — stop after N, record the window, come back — which is a structural decision made long before the outage.

The fix#

Decide routing explicitly, per task class, and write it down where the dispatcher reads it:

# Routing is a DESIGN decision, recorded per task class rather than chosen per invocation.
# "Which engine should run this?" asked fresh each time gets answered by whatever is nearest.
routine-commit:        deterministic-script     # enumerable procedure — no model
rebase-conflict:       reasoning                # genuinely ambiguous
verify-audit:          cheap-model              # read-only, schema-shaped output
codegen:               capable-model + warm     # needs the trained context, loaded ONCE

Three properties make this work rather than just documenting an intention:

The default is the script. Reasoning is the exception that has to be justified, not the fallback. Reverse that and every routine task drifts toward the expensive path, because the expensive path also works.

Context is scoped per class and the omissions are declared. A worker asserts the surfaces it needs; a missing one halts rather than being improvised around. Saving 12.6% matters far less than knowing what was dropped.

Retries terminate. Every loop against an external service has a ceiling and a recorded outcome. Without one, a throttling window becomes an unbounded bill.

How we proved it#

Not by comparing monthly invoices — too many variables, too slow a signal.

The routing claim was measured directly: the same three-repository commit, both ways. The script path completes in seconds with no model call. The reasoning path cost ~180,000 tokens and 7½ minutes. Same output, both correct.

The context claim was measured by building the pack both ways and diffing the byte count: 464,099 → 405,506, about 14,600 tokens per worker. And the measurement corrected the estimate downward by more than a factor of three — the pre-measurement guess was 40%. Had nobody measured, a plausible number would have been quoted in a design document and believed.

The retry claim is today's log: 16 calls, 25 minutes, zero outputs, escalating backoff. The fix is not a better prompt; it is a loop that stops.

What none of this establishes is a general ratio. These are three measurements in one codebase, and the order of the levers is what I would expect to transfer, not the magnitudes. A codebase whose work is genuinely ambiguous will route far more to reasoning and be right to.

Where this is wrong#

Two limits, and the first one is a trap I nearly walked into while writing the routing table.

"Default to the script" degenerates into scripting everything. The reason a model was reachable for that commit at all is that the hard cases are real: a rebase conflict, a repository in a state nobody enumerated, a halt that needs a judgement call. Those are exactly the cases a script handles by failing, and the failure lands on a human at the worst moment. The rule is default to the script, not only the script — and the test is whether the procedure can be enumerated, not whether it usually goes smoothly.

Routing has its own cost, and it is paid in staleness. A routing table is a second thing to maintain. When a task class changes shape and the table does not, work gets dispatched to an engine that no longer suits it — and unlike a bad prompt, nothing about the output says so. The mitigation is the same as everywhere else in this post: the table is read by the dispatcher rather than consulted by a human, so a class with no entry fails loudly instead of defaulting to whatever is nearest.

What this generalizes to#

Ask the cost question one level up from where it is usually asked.

"Which engine should run this?" beats "how do I make this call cheaper?" A correct answer produced by the wrong engine is the most expensive kind, because it never looks like a problem.

Measure the context, do not estimate it. The estimate here was wrong by 5×, in the direction that would have justified a bigger change than the evidence supported. Context is easy to weigh and nobody weighs it.

Treat parallelism as a cost with a latency benefit, which is what it is. Fan-out multiplies per-worker context. Sometimes worth it; never free.

Give every external loop a ceiling. An outage with unbounded retries is a bill with no upper bound, and the terminating condition has to exist before the outage does.

The uncomfortable part is that none of these are prompt-engineering skills. They are ordinary architecture — routing, scoping, caching, backoff — applied to a component that happens to be a model. The industry is spending its attention on the wording, and the bill is being set by the wiring.

Related#

• Where your AI coding bill actually goes — the measured breakdown of one long session
• One warm context beats five cold ones — the third lever, at length
• Not every step needs the expensive model — routing as a tier decision

Keep reading#

This is part of Agent tooling & workflows — the engineering around the model, which is where most of the difficulty actually lives. Follow MobileByteSensei for the rest.