Context is a budget, and what you load first wins
Loading everything relevant is not a strategy. Measure tokens per decision.
TL;DR
A large context window invites you to load everything plausibly relevant. We did: 180,000 tokens of guides, conventions, type definitions and neighbouring files per code-generation call.
Then we measured which of it changed an output. Removing 60% of the context left the generated code identical on 52 of 60 tasks, improved it on 3, and degraded it on 5. The 60% was not helping; it was being paid for.
Context is a budget with a cost per call, and the useful question is tokens-per-decision: for each block you load, does removing it change what comes out? The answer is measurable, and it is usually no.
The case this post is about
A code generator assembling its context the obvious way — everything that might matter:
def build_context(feature: str) -> str:
return "\n\n".join([
read("docs/ARCHITECTURE.md"), # 14k tokens
read("docs/CONVENTIONS.md"), # 9k
read("docs/TESTING_GUIDE.md"), # 11k
read("docs/DESIGN_SYSTEM.md"), # 16k
*[read(p) for p in all_type_definitions()], # 38k
*[read(p) for p in neighbouring_features()], # 71k (3 similar features, in full)
read(f"specs/{feature}.yaml"), # 4k
git_log_summary(limit=50), # 6k
read("CHANGELOG.md"), # 11k
]) # ≈ 180k tokens
Reasonable-sounding: the model should know the architecture, the conventions, the design system, and what similar features look like.
The symptom
Cost and latency that made the agent impractical:
per generation call
input tokens 181,400
cost $0.54
latency 38s
Across a 60-feature implementation run: $32 and 38 minutes of pure prefill.
And a quality problem that was the actual surprise. The generator started producing code patterned on the wrong neighbouring feature:
task: implement the `reminders` feature (a simple CRUD list)
generated: a Store5-backed paginated repository with a Bookkeeper and
conflict resolution — copied from the `sync` feature in the context
The sync feature was 31k tokens of the 71k neighbouring-features block. It was the largest,
most detailed thing in the context, and it dominated the output for tasks it did not fit.
A third symptom, from a lost-in-the-middle check: the specs/{feature}.yaml — the single most
important input — sat at roughly position 155k of 180k. Moving it to the front changed the output on
11 of 60 tasks, all improvements.
Why the obvious fix failed
"The window fits it, so there is no problem." Fitting is not the same as attending. The spec being in the window did not mean it was weighted against 71k tokens of example code.
"Summarise the big documents." We compressed the four guides from 50k to 12k with a summarisation pass. Outputs changed on 9 of 60 tasks, 6 for the worse — the summaries dropped the specific conventions that mattered and kept the general advice that did not. Lossy compression of a reference document loses the parts that are referenced.
"Use retrieval instead of loading everything." The right direction, and our first version retrieved by embedding similarity and reliably returned the test file for whatever it was asked about — which is its own article. Retrieval quality became the new problem.
"Just use a bigger model with a bigger window." Addresses the fitting problem we did not have, at a higher price per token.
The actual mechanism
Three separate effects, and conflating them is why "load more" feels safe.
Cost and latency scale with input tokens. Linear, immediate, and the easiest to measure.
Attention is finite and position-sensitive. Models attend unevenly across a long context — the beginning and end are weighted more heavily than the middle. A critical instruction at position 155k competes poorly with 71k tokens of concrete code earlier.
And concrete, detailed content outcompetes abstract guidance. This is the one that produced our quality bug: a full implementation of a similar feature is a far stronger signal than a sentence in a conventions document saying "prefer the simplest repository that works". The model pattern-matches on what is available, and 31k tokens of Store5 code is very available.
Which gives the measurable question:
For each block in your context: if you remove it, does the output change? If not, you are paying for it. If it changes for the worse, it is load-bearing. If it changes for the better, it was actively harmful.
The sub-case that decides whether you can trust the result: ablation measures the task distribution you ran it on. A block that is identical across sixty tasks is not proven useless — it is proven useless on those sixty. Guidance documents are frequently insurance against the rare case, and the rare case is exactly what a sample of common tasks does not contain.
The conventions file we cut is the honest example. It made no difference across sixty tasks, because fifty-eight of them touched features where the conventions were already visible in the surrounding code. The two where it would have mattered — a new module with no neighbours to imitate — were not in the sample, because new modules are rare, which is why someone had written the conventions down.
So the rule is narrower than "ablate and delete":
- Identical across the sample → it is not earning tokens on routine work. Move it behind a trigger rather than deleting it outright.
- Better without → delete it. Actively harmful needs no rare-case defence.
- Worse without → load-bearing, and now you know.
The middle category is the whole finding. The bottom one is a measurement. The top one is a question about your sample, and answering it with "delete" is how a team removes the document that was holding up the case nobody tested.
The fix
Ablate every block, keep what earns its tokens, and put the important thing first.
1. The ablation, run once as an experiment:
BLOCKS = ["architecture", "conventions", "testing_guide", "design_system",
"type_defs", "neighbours", "spec", "git_log", "changelog"]
for block in BLOCKS:
for task in TASKS_60:
baseline = generate(task, context=full_context(task))
ablated = generate(task, context=full_context(task, omit=block))
record(block, task, identical=(normalize(baseline) == normalize(ablated)),
better=judge_which_is_better(baseline, ablated, task))
block tokens outputs identical better without worse without
changelog 11k 60/60 0 0
git_log 6k 60/60 0 0
architecture 14k 58/60 1 1
design_system 16k 57/60 0 3
testing_guide 11k 49/60 2 9
conventions 9k 41/60 1 18
type_defs 38k 12/60 0 48
neighbours 71k 31/60 17 12
spec 4k 0/60 0 60
Reading that table bottom-up is the whole fix. spec and type_defs are load-bearing.
conventions and testing_guide earn their keep. changelog and git_log changed nothing, on
every task. And neighbours — the largest block — was better removed 17 times and worse 12
times, which is barely better than noise for 71k tokens.
2. The rebuilt context, ordered deliberately:
def build_context(feature: str) -> str:
spec = read(f"specs/{feature}.yaml")
return "\n\n".join([
# the task, FIRST — strongest attention position
f"<task>\n{spec}\n</task>",
# load-bearing reference
read("docs/CONVENTIONS.md"), # 9k
read("docs/TESTING_GUIDE.md"), # 11k
*[read(p) for p in types_referenced_by(spec)], # ~9k, not all 38k
# ONE example, chosen by archetype match, and trimmed to its shape
skeleton_of(nearest_feature_by_archetype(spec)), # ~6k, not 71k
# the task again, LAST — the other strong position
f"<task-reminder>\n{spec.summary}\n</task-reminder>",
]) # ≈ 39k tokens
Four decisions:
The spec goes first and is restated last. Both high-attention positions, for the one input that mattered on 60 of 60 tasks.
Type definitions are filtered to those the spec references. 38k to 9k, because the ablation showed type defs are load-bearing as a class — not that all of them are needed for every task.
One neighbour, matched by archetype, reduced to a skeleton. Signatures and structure, not bodies:
def skeleton_of(path: Path) -> str:
# keep declarations, drop implementations — the shape is the signal
return strip_function_bodies(read(path))
This is what fixed the Store5 bug: a skeleton conveys "this is how a feature is laid out" without being a 31k-token template to copy.
And changelog and git_log are deleted. Zero effect on 60 tasks is a clear enough result.
3. Then keep measuring, because context accretes the same way dead code does:
# in CI, monthly: re-run the ablation on a 10-task sample
# any block with 10/10 identical outputs is flagged for removal
How we proved it
The headline numbers:
before after
input tokens 181,400 38,900
cost per call $0.54 $0.12
latency 38s 11s
60-task run cost $32 $7
That quality did not drop — the claim that matters, since the easy version of this is to cut context and ship worse code:
60 tasks, hidden-test pass rate
full context (181k): 47/60 (78%)
trimmed (39k): 51/60 (85%)
Better, by 4 tasks. Three of those four were the Store5-pattern bug no longer firing.
The position effect, isolated:
spec position in context tasks where output changed vs spec-first
position 155k (original) 11/60 (all 11 worse)
position 0 (first) baseline
first AND restated last 3/60 (2 better, 1 neutral)
And the skeleton-versus-full-neighbour comparison:
neighbour block form tokens pass rate wrong-archetype copies
full implementation 71k 78% 8
skeleton only 6k 85% 1
omitted entirely 0 82% 0
The middle row is why the skeleton stayed rather than being cut: it beats both extremes. Omitting the neighbour entirely loses structural guidance; including it in full imports the wrong pattern.
Two honest limits. The ablation is per-block, so it cannot detect interactions — two blocks that are each individually removable but jointly load-bearing would show as removable. We spot-checked the three removals together and saw no degradation, which is evidence rather than proof. And these results are for one task type on one model; the method transfers and the specific table does not.
What this generalizes to
Measure the marginal value of every input. Ablation is the method: remove one thing, see if the output changes. It applies to prompt sections, retrieved documents, features in a model, fields in a request, and columns in a report — and the usual finding is that a large fraction changes nothing.
Position matters where attention is uneven. The most important instruction belongs where it will be weighted most. Measuring this takes one experiment and is worth doing once per pipeline shape.
Concrete content outcompetes abstract guidance. A full example will beat a sentence of advice every time. When you include an example, you are choosing a template — so choose it deliberately, and consider whether its shape is sufficient.
A context window is a budget, not a container. "It fits" says nothing about cost, latency, or whether the model attended to the part you cared about. Treat every token as purchased.
Related
- The retrieval worked and returned the wrong file — the retrieval alternative, and its own failure
- Your prompt is a config file, and it needs the same discipline — versioning what you load
- The most valuable pull request that quarter deleted 4,000 lines — the same accretion problem, in code
Keep reading
This is part of Context engineering for code — what to put in a model's context, measured rather than assumed. Follow MobileByteSensei for the rest.
One post a week, no digest
New writing when it lands. No roundups, no course pitch, no "10 tips".