Say what you truncated or the report is a lie

A scan reports eighteen issues across the project. It is accurate, well organised, and the eighteen are real.

It examined the first fifty files of two hundred and forty.

The symptom#

A report whose scope is invisible. Everything in it is true; what is missing is any statement about what was looked at, and a reader supplies the natural assumption — that the report covers the thing it is about.

The second thing is how many ordinary engineering decisions produce this. A top-N limit to keep output readable. A sample because the full set is slow. A timeout that ended a pass early. A no-retry policy that dropped the items that failed once. Each is defensible, each is invisible in the result, and each turns a complete-looking report into a partial one.

The third is that the bound is usually known precisely at the moment it is applied and nowhere afterwards. The code that slices the list knows exactly how many it dropped. By the time the summary is written, that number has not been carried forward, and reconstructing it means re-reading the implementation.

The fourth is the asymmetry in how the error is read. An over-reported number gets challenged — a scan claiming four hundred issues invites scrutiny. An under-reported one is read as good news, and good news is not audited.

That asymmetry is why these survive. A silent cap makes a system look healthier than it is, and nothing in an organisation is set up to question a result in that direction.

There is a fifth property that makes this particularly stubborn in automated systems: the bound is often applied by something that has no idea a report will be written from its output. A function that returns the first fifty results is behaving correctly; the misreporting happens two layers up, where nothing knows a slice occurred.

Why the obvious fix failed#

Document the limits. Puts the bound somewhere other than where the number is read. The report travels — into a message, a dashboard, a summary of a summary — and the documentation does not travel with it.

Log the truncation. Correct and ineffective for the same reason. The log is read when something has gone wrong, and nothing has gone wrong.

Remove the caps. Sometimes right and usually not available. Caps exist for reasons — cost, runtime, output size — and removing them trades a reporting problem for a resource problem.

Make the cap large enough not to matter. The compromise that feels safe. It converts a cap that fires often into one that fires rarely, which makes it less visible rather than less real, and the rare case is now the one nobody is prepared for.

Sort so the important things come first. A genuinely good idea that solves a different problem. Ranking makes a truncated list more useful and does nothing about the reader's belief that the list is everything — and it adds a second silent assumption, that the ranking is right.

The actual mechanism#

Put the bound in the output, next to the number, every time one is applied.

examined   how many items the run actually looked at
total      how many there were
dropped    what was excluded, and by which rule

The important property is that this travels with the result. A number carrying its own denominator cannot be quoted out of context into a summary that implies completeness, because the context is part of the value.

It also changes what a reader does. "Eighteen issues" invites a decision; "eighteen issues in the fifty files examined, of two hundred and forty" invites a question, and the question is the correct response to a partial result.

The sub-case that is easiest to get wrong is the cap that did not fire. A run that examined everything should say so explicitly rather than omitting the scope line, because an absent line is ambiguous — it could mean full coverage or it could mean somebody forgot. Stating the scope unconditionally is what makes its presence meaningless and its contents meaningful.

The alternative — printing the scope only when something was dropped — seems tidier and quietly recreates the problem. A reader who has never seen the line cannot distinguish a run with full coverage from a run whose scope reporting was never wired up, and both look like silence.

A related one: the cap several layers down. A summary assembled from component reports inherits every bound its components applied, and unless each one carries its scope, the aggregate silently reports the union of several partial views as though it were a whole. Scope has to compose, which means it has to be data rather than prose.

Prose scope is the version that usually gets built first, because it is what a report looks like to a person writing one. It reads well and cannot be summed, so the moment two reports are combined the information is gone — and combining reports is exactly when it was going to matter.

The fix#

carry     every bounded operation returns its bound alongside its result
state     scope appears in the output unconditionally, firing or not
compose   aggregates combine scopes rather than discarding them
name      say which rule dropped what — timeout, sample, top-N, no-retry

The four lines as one type, which is the only way the third one is achievable:

/** A result that cannot be quoted without its denominator, because they are one value. */
interface Scoped<T> {
  value: T;
  examined: number;
  total: number;
  dropped: Array<{ count: number; rule: string }>;   // NAMED, not just counted
}

/** Composition is the whole point: an aggregate inherits every bound beneath it. */
function merge<T>(parts: Array<Scoped<T[]>>): Scoped<T[]> {
  const dropped = new Map<string, number>();
  for (const p of parts) {
    for (const d of p.dropped) dropped.set(d.rule, (dropped.get(d.rule) ?? 0) + d.count);
  }
  return {
    value: parts.flatMap((p) => p.value),
    examined: parts.reduce((n, p) => n + p.examined, 0),
    total: parts.reduce((n, p) => n + p.total, 0),
    dropped: [...dropped].map(([rule, count]) => ({ rule, count })),
  };
}

/** Rendered UNCONDITIONALLY. The full-coverage case says so rather than staying silent. */
function render<T>(r: Scoped<T[]>): string {
  const head = `${r.value.length} finding(s)`;
  if (r.examined === r.total && r.dropped.length === 0) {
    return `${head} · examined all ${r.total}`;
  }
  const why = r.dropped.map((d) => `${d.count} ${d.rule}`).join(", ");
  return `${head} · examined ${r.examined} of ${r.total} · not examined: ${why}`;
}

And the two reports it produces, which is the comparison the whole post is about:

$ deno task audit --report
18 finding(s) · examined 50 of 240 · not examined: 148 per-item-timeout, 42 top-N-cap

$ deno task audit --report --no-limits
31 finding(s) · examined all 240

The first line is the same run that used to print 18 findings. Nothing about the audit changed — only whether the bound travelled with the number — and the second run is what the first was being read as.

Note that render has no branch for "scope reporting not wired up", because the type makes that state unrepresentable. That is the part retrofitting cannot reproduce: a chain where one layer returns a bare T[] has to reintroduce the ambiguity somewhere, and the honest place is a dropped: [{ rule: "unknown-upstream-bound" }] entry rather than an omitted line.

The fourth line is what makes a partial report actionable rather than merely honest. "Forty items not examined" prompts a shrug; "forty items not examined — exceeded the per-item timeout" is a finding in its own right, and often a more interesting one than anything in the report proper.

The hardest part in practice is the third. Scope composition requires every layer to participate, and one layer that returns a bare number collapses the whole chain back to ambiguity — which means this is a property of a system rather than of a report, and it has to be built in at the bottom.

Retrofitting it is correspondingly awkward. There is no useful partial adoption: a chain where four of five layers carry scope reports the same ambiguity as one where none do, so the work only pays at the end, which makes it a hard thing to start.

How we proved it#

The gap is measurable the first time you look. Instrumenting a scan to report examined-versus-total and comparing against what it had been reporting is a single change, and the answer is either reassuring or it is not. In our case it was not: a media generation step covered a fraction of what its summary implied, and the summary was green.

The decision changed once the denominator was visible, which is the practical argument. The same eighteen findings, with the scope attached, produced a different next action — go look at the rest — where before they had produced a sense of completion.

What I have not measured is how many bounded operations in the system still report bare numbers. The ones I know about are the ones I have been through; the rest are unknown, and the honest position is that any number without a denominator might be one of them.

The compose failure is the one I would look for first in any existing system, because it is both common and invisible from the top. Each component can be scrupulous about its own scope and the aggregate still reports a bare total, and the layer that discarded the information is usually the most innocuous-looking one in the chain.

What I have not solved: the bound that is not countable. A search that may simply not have found something has no denominator to report — there is no total to divide by — and for those the honest statement is about method rather than coverage, which is weaker and much harder to make useful.

The best I have managed there is to state what was searched and how, so a reader can judge the method even when there is no coverage figure to give. It is weaker evidence and it is at least evidence of the right kind — about the search rather than about the searcher's confidence.

And a limitation worth naming: stating scope makes a report honest, not complete. A partial scan that says it is partial is still a partial scan, and there is a real risk that the scope line becomes boilerplate — present, correct, and skimmed past — at which point it is documentation again.

The thing that reframed it was noticing that a report's shape encodes a claim nobody wrote. A list of findings about a project asserts, by its form, that it is about the project; stating the scope is not adding information so much as correcting a claim the format was making on its own.

Which is also why this cannot be fixed by being careful in the summary. The implication comes from the structure, so the correction has to be structural too — the denominator has to be part of the value, not an adjacent sentence somebody remembered to write.

What this generalizes to#

Any bounded operation produces output shaped like an unbounded one unless the bound is in the result.

The pattern is everywhere: a query with a limit clause feeding a count, a paginated API consumed without following the pages, a log search over a default time window, a test suite with a filter left on. In each case the system behaves exactly as asked and the answer is about a subset nobody mentioned.

So the question worth asking of any number is what was its denominator, and did anything get dropped on the way. If the answer is not in the output, it is not in anybody's reasoning either — and the number will be used as though the answer were "nothing".