A generator given the wrong context does not stop

A code generator is handed a pack of reference material before it starts: the conventions, the worked examples, the guides for every layer it might touch. The pack is assembled by a glob.

A glob that misses a file does not report anything. It returns the files it matched.

The case this post is about#

One generator, about to write a feature module, asserting the reference surfaces it needs:

exit 0    present
exit 3    deliberately excluded for this project type
exit 4    undeclared — the generator wants something nobody listed
exit 5    the pack was never built

Exit 3 and exit 4 are the whole design. A surface absent because this project type never reads it is the scoping working correctly. A surface absent because no glob ever matched it is a generator about to improvise, and before those two codes existed they were the same condition.

That is not hypothetical: a guide the generators genuinely needed had never been in the pack at all, because the glob that assembled it did not match the filename. This post follows that one assertion, because the dangerous pack is not the bloated one.

The symptom#

The generated code is plausible and unlike the codebase. Not wrong in a way that fails to compile — wrong in the way a competent stranger would be wrong, inventing a reasonable convention because the actual one was not in front of them.

The second thing is that nothing reports a gap. The pack was built, the generator ran, the output appeared. A missing reference is not an error condition anywhere in that chain: the assembling step found what it found, and the generating step used what it was given.

The third is that the improvisation is confident and locally consistent. A generator that has no example of how a particular layer is written will produce something coherent, apply it uniformly across every file it emits, and leave you with a dozen files that agree with each other and with nothing else in the repository.

The fourth is the one that took me longest to see. I was looking for the pack to be too big — too much irrelevant material, costing tokens and diluting attention — because that is the failure everyone writes about. The pack being too small has no symptom at all until you read the output carefully, and by then there is a dozen files of it.

There is a reason that asymmetry persists. A bloated pack has a number attached to it — bytes, tokens, cost per run — so it shows up in exactly the place people look when they think about context. A thin pack has no number at all. Nothing in any dashboard goes up when a surface is absent, so the only instrument is somebody reading the output and recognising a convention that is not the house one.

Why the obvious fix failed#

Put everything in the pack. The safe-seeming default and it is genuinely expensive: the pack is loaded per subagent, so irrelevant material is paid for on every single invocation. It also means a generator for one project type carries guides that only a different type's generators ever read.

Scope the pack by project type. The right move and it is where the real risk enters. Removing material is exactly the operation that can remove something needed, and the removal is silent in precisely the same way the original glob miss was.

Estimate the saving to justify the work. I guessed around forty per cent. The measured figure was 8.2% — about 6.2k tokens per subagent, from 305,942 bytes down to 280,963. Still worth having on a fixed per-invocation cost, and not the number I would have defended in a planning conversation if nobody had measured it.

Check the pack looks right. Reading a file listing tells you what is there. The failure is about what is not there, and absence is not something you notice in a list you are scanning for confirmation.

The actual mechanism#

Make the generator assert the surfaces it needs, and make a missing one a hard stop rather than a silent improvisation.

build     the pack, scoped by project type, stamping what was included and excluded
require   each generator declares the surfaces it depends on, by name
exit 0    present
exit 3    deliberately excluded for this project type
exit 4    undeclared — the generator wants something nobody listed
exit 5    the pack was never built

The distinction between exit 3 and exit 4 is the whole design. "Excluded on purpose for this type" and "nobody ever added this" are completely different conditions that look identical from inside a generator, and separating them is what makes a thin pack safe to ship: the first is a decision you can read, the second is a gap that halts the run.

The sub-case worth stating plainly is that the glob is the actual culprit, not the scoping. One guide was never in the pack at all — its filename did not match the pattern that assembled the others, so it had been absent since the pack was introduced, and nothing had ever asked for it by name. Scoping by type did not cause that; it only made me look.

There is a related property that falls out for free. Once the pack records what it included and excluded, two generators running for different project types can no longer share a pack path and clobber each other — which was a real race condition before and was invisible because the symptom was, again, output that looked slightly wrong.

That pattern repeats often enough to be worth noticing on its own: several distinct defects here — the glob miss, the shared path, the under-declared dependency — all present as the same vague symptom. When a system's only failure signal is "the output seems a bit off", the number of distinct causes hiding behind it is usually more than one.

The fix#

scope      the pack per project type, at a per-type path
stamp      pack-includes and pack-excludes, in the pack itself
assert     every generator names its required surfaces before emitting anything
halt       on undeclared or unbuilt, never proceed, never read around it

All four lines, in the order they run. The excluded list is what lets the assert tell two very different absences apart:

type Surface = { id: string; path: string; types: string[] };

export async function buildPack(type: string, out: string) {
  const included: string[] = [], excluded: string[] = [];
  let body = "";
  for (const s of SURFACES) {
    if (!s.types.includes(type)) { excluded.push(s.id); continue; }   // scoped out
    body += `\n\n<!-- ${s.id} -->\n` + await Deno.readTextFile(s.path);
    included.push(s.id);
  }
  await Deno.writeTextFile(`${out}/pack.${type}.md`, body);
  await Deno.writeTextFile(`${out}/pack.${type}.json`,
    JSON.stringify({ type, included, excluded, bytes: body.length }, null, 2));
}

export function requireSurfaces(declared: string[], manifest: string): number {
  let m: { included: string[]; excluded: string[] };
  try { m = JSON.parse(Deno.readTextFileSync(manifest)); }
  catch { console.error("pack not built"); return 5; }

  const missing = declared.filter((id) => !m.included.includes(id));
  if (missing.length === 0) return 0;

  const undeclared = missing.filter((id) => !m.excluded.includes(id));
  if (undeclared.length === 0) {
    console.error(`excluded for this type: ${missing.join(", ")}`); return 3;
  }
  console.error(`undeclared surface: ${undeclared.join(", ")}`); return 4;
}

Three runs against a pack built for one project type:

$ require CORE_STORE CORE_DATA
exit 0

$ require CORE_STORE GAME_SCENE          # exists, scoped out for this type
excluded for this type: GAME_SCENE
exit 3

$ require CORE_STORE CORE_NAV            # in neither list — nobody declared it
undeclared surface: CORE_NAV
exit 4

Exit 4 is the one worth having. Without it the generator produces the layer anyway, from general knowledge rather than this codebase's conventions, and it compiles.

The fourth line needs saying because the obvious workaround defeats the entire mechanism. A generator that cannot find a surface in its pack must not go and read the file directly — that restores the original condition with an extra step, and the assertion becomes decorative.

What makes this affordable is that the assertion is a path check, not an analysis. Each generator names the surfaces it uses, something compares that list against what the pack contains, and the result is one of four exit codes. There is no model in the loop and nothing to tune.

It is also fast enough to run at the top of every generator rather than once per build. That matters more than it sounds: a check at the start of a run halts before anything is written, so the failure costs nothing to recover from, whereas the same check at the end leaves a directory of improvised files to clean up.

How we proved it#

The saving is measured and smaller than I claimed. 305,942 bytes to 280,963, about 6.2k tokens per subagent, 8.2%. I include the number I guessed — forty per cent — because the gap between them is the useful part: a plausible estimate about context size was off by a factor of five, and the only reason I know is that somebody measured instead of agreeing with me.

The estimate was not careless, which is the uncomfortable part. It came from eyeballing which guides were irrelevant to a given project type and assuming irrelevant meant large. The guides that could be dropped turned out to be the short ones, and the bulk was in material every type needs — which is obvious in hindsight and was invisible before the measurement.

The missing guide is a demonstrable glob failure. The pattern that assembled the pack matched a naming convention that one guide did not follow. Listing the pack's contents against the directory it was assembled from shows it immediately, and it had been absent for the entire life of the pack.

What I cannot tell you is how much output it affected. The guide that was missing covers a specific build area, and I have not gone back through generated files attributing improvisations to its absence. The honest position is that the mechanism was broken for months and I do not have a damage figure.

What I have not solved: deciding what a generator actually needs. The assertion checks a declared list against the pack; nothing checks that the declared list is right. A generator that under-declares will pass its assertion and improvise exactly as before, and working out the true dependency set means reading the generator's prompt with care — which is a human job I have done once and will have to do again every time one changes.

What would close that gap is deriving the declaration rather than writing it — having the generator's required surfaces fall out of its prompt automatically, so the two cannot disagree. I have not built it, and it is the same shape as every other derive-rather-than-maintain problem: the hand-written list works until somebody edits one half.

And a cost that is real: per-type packs mean more packs, each with its own staleness question. A content edit to any source file invalidates that type's pack, and now there are several to keep current rather than one. The invalidation is automatic; the multiplication of things that can be stale is not nothing.

The thing that reframed it was noticing which direction I had been worrying in. Every instinct about context is about excess — too much, too noisy, too expensive — because excess is what you can see in a token count. Deficiency is invisible in every measurement I had, and it is the one that changes what gets written.

It also explains why the fix had to be an assertion rather than a better pack. No amount of care in assembling the pack addresses a generator needing something nobody thought to include — only the consumer knows what it requires, so only the consumer can report that it is missing.

What this generalizes to#

A missing input produces output, and that is the whole problem.

A system that fails loudly on a missing input is easy to reason about. The dangerous ones are those where absence is indistinguishable from a legitimate value: a configuration key that defaults, a glob that returns fewer files, an optional parameter nobody passed, a feature flag read before it was created. In each case something sensible happens and nothing records that the sensible thing was a substitute.

So the question worth asking of any assembled input is how would this look if a piece were missing — and if the answer is "the same, with slightly different output", then the assembly step needs to state what it expected, not just deliver what it found.

The cheapest version of that is the one I would reach for first anywhere: have the assembling step emit its own manifest — what it looked for, what it found, what it deliberately left out. None of those three is expensive to record, and together they turn a silent absence into a line somebody can read.