Partial translation is a number, not a feeling

An app supports six languages. Somebody asks how complete the Spanish is, and the answer that comes back is a guess — probably most of it, the main screens are done, there might be some gaps in settings.

That answer is available as a number, exactly, for every locale, in about a second.

The case this post is about#

One question, asked before a release: is Hindi finished?

base keys        the keys in the default locale
locale keys      the keys present and non-empty in locale X
coverage         |locale ∩ base| / |base|
missing          base \ locale, listed by key

That question has an exact answer and nobody was computing it. What happened instead was that somebody opened three screens, saw Hindi on all three, and said yes.

The arithmetic is four lines. This post follows that one question, because two details in it decide whether the number is usable: a key that is present but empty is missing, and a key that is deliberately never translated has to come out of the denominator — or every locale is capped below 100 per cent forever, and a number nobody can reach is a number nobody reads.

The symptom#

The visible failure is a language shipped in a state nobody would have approved if they had seen it stated. Not catastrophically broken — eighty per cent translated, with the remaining fifth scattered across error messages, empty states and newer features, which are the screens nobody walks through when sanity-checking a release.

The second thing is who ends up deciding. With no number, "is Spanish ready" is answered by whoever last opened the app in Spanish, which is usually the person who added the Spanish strings, who is looking at the screens they just worked on. The sample is biased by construction and the conclusion is reported with reasonable confidence.

The third is the drift between releases. Every feature added in English adds keys to the base set, and every one of those is a new gap in all six locales. A language that was complete in March is not in June, and nothing announced the change — the coverage fell because the denominator grew, which is invisible from inside any single feature's work.

The fourth is the asymmetry of discovery. Missing translations are found by speakers of that language, who are usually users rather than the team, and who report them one string at a time if at all. So the feedback arrives slowly, incompletely, and in a form that makes the problem look smaller than it is — three reports about three strings, when the real number is ninety.

Why the obvious fix failed#

Check it before each release. Means opening the app in each language and walking the screens, which is hours per locale and gets skipped under pressure. It is also unreliable — a walk-through covers the paths somebody thought to walk, and the gaps concentrate in the paths nobody does.

Ask the translator whether it is complete. They can only speak to what was sent to them. If the export omitted keys — because they were added after the last export, or because something was never a resource — the translator's "yes, complete" is accurate about their file and wrong about the app.

Watch for untranslated strings in review. A reviewer sees a diff adding English strings to the base set, which is correct and expected. The absence of corresponding entries in five other files is not in the diff, because nobody edited those files — and that absence is the entire problem.

Set up continuous translation. The right long-term answer and it does not remove the need for the number. An automated pipeline still has a lag, still has strings in flight, and still leaves you needing to know where a locale stands at the moment you are deciding to ship.

The actual mechanism#

Coverage is a set comparison, and both sets are already on disk.

base keys        the keys in the default locale
locale keys      the keys present and non-empty in locale X
coverage         |locale ∩ base| / |base|
missing          base \ locale, listed by key

That is the whole computation. It needs no service, no integration and no process — the resource files contain everything required, and the result is exact rather than estimated. The reason it is usually absent is not difficulty; it is that nobody framed the question as countable, so it stayed a matter of opinion.

Three details make the number trustworthy rather than merely available.

Present-but-empty counts as missing. A key that exists in the locale file with an empty value is structurally present and functionally absent, and a naive key-set comparison scores it as covered. That single detail is the difference between a number that means something and one that is reliably optimistic.

Report per locale, not aggregate. An average across six languages tells you nothing actionable — it hides one language at forty per cent behind five at ninety-five. The useful output is a row per locale, and the decision is made per locale too.

The same reasoning applies one level down, and it is where I would resist adding detail. A per-screen or per-feature breakdown is tempting and mostly produces a table nobody reads: the actionable unit is a locale, because that is what gets shipped or not. The key list already tells you where the gaps are for anybody who needs it, and it does so without inventing a second hierarchy to maintain.

Compute it per release, not on request. A number that is calculated when somebody remembers to ask is a number that is calculated after the decision has effectively been made. Produced automatically at release time, it arrives before the question does.

The sub-case worth handling explicitly: keys that should not be translated. Brand names, format specifiers, technical identifiers. Counted as missing they permanently depress every locale's percentage, and a metric that can never reach a hundred stops being read. Marking them excluded is one annotation on the key and it is what keeps the ceiling meaningful.

The fix#

compute   per locale: covered, missing, and the missing keys by name
exclude   explicitly non-translatable keys, marked at the key
empty     treat present-but-empty as missing
report    per release, automatically, as a row per locale

The two rules that make the number honest are both in the counting, not in the reporting — an empty string is missing, and a non-translatable key is out of the denominator entirely:

type Coverage = { locale: string; covered: number; total: number; missing: string[] };

export function coverage(base: Map<string, string>, locales: Map<string, Map<string, string>>,
                         nonTranslatable: Set<string>): Coverage[] {
  // Denominator EXCLUDES keys that are deliberately untranslated. Leaving them in
  // caps every locale below 100% forever, and a number nobody can reach is a number
  // nobody reads.
  const keys = [...base.keys()].filter((k) => !nonTranslatable.has(k));

  return [...locales].map(([locale, m]) => {
    const missing = keys.filter((k) => {
      const v = m.get(k);
      return v === undefined || v.trim() === "";   // present-but-EMPTY is missing
    });
    return { locale, covered: keys.length - missing.length, total: keys.length, missing };
  }).sort((a, b) => a.covered / a.total - b.covered / b.total);
}

Per release, a row per locale, with the missing keys named rather than counted:

$ deno task locale-coverage
  locale   covered        missing
  de        412/486  85%  profile_save, profile_delete_confirm, error_offline, …(74)
  fr        486/486 100%
  es        481/486  99%  settings_export_title, settings_export_body, …(5)
  excluded      12         brand_name, app_id, currency_symbol, …

German's 85% is the useful output; the five missing in Spanish are the actionable one. Counting empty strings as present would have reported de at 97%, because a translator had committed the keys with nothing in them.

The third line is the one that is easy to skip and is where most naive implementations lose their credibility. The fourth is what turns the number into a decision input rather than an audit artifact.

What this buys is not better translations. It is that "should we ship Spanish" becomes a question with a stated answer, made by somebody looking at ninety-one per cent and a list of what is missing, rather than by somebody recalling that the main screens looked fine.

The list of missing keys matters as much as the percentage, and it is the half that gets dropped because a single number is easier to put in a report. A percentage tells you whether to worry; the key list tells you what the work is, and it turns "Spanish is at ninety-one" into forty named strings that somebody can be given. Without it the number is a verdict with no next action, which is how metrics end up being quoted and ignored.

There is also a decision about what the number is for, and it is worth making explicitly before anybody has an opinion about the figure. A coverage threshold used as a release gate behaves very differently from one used as information: a gate invites the lowest-effort route to clearing it, which for translations is machine output pasted into the gaps. Published as information, alongside the key list, the same number tends to get acted on properly, because nothing is standing between somebody and shipping except their own judgement about whether the gaps matter.

How we proved it#

The computation is exact and verifiable, which is unusual for anything describing itself as coverage. Both sets are finite, both are on disk, and the comparison has no sampling, no heuristic and no tuning. A disagreement about the number is a disagreement about the files, which can be settled by looking.

The empty-value detail is demonstrable: add a key to a locale with an empty string, and a key-set comparison reports it covered while the app shows nothing. That takes a minute and it is what justifies treating empty as missing rather than arguing about it.

What I cannot claim is that the number changed any shipping decision. I built it because the question was being answered by impression, not because a bad release had already happened — so the honest framing is that it replaced a guess with a fact, and I have not yet had the case where the fact stopped something.

What I have not solved: quality. Coverage counts entries and says nothing about whether a translation is correct, idiomatic, or the right length for its control. A locale at a hundred per cent can be full of machine output that reads badly, and the number will report it identically to one that was carefully done. I have been careful to describe this as coverage rather than readiness for exactly that reason.

And a limitation on what the denominator means: coverage is measured against the base key set, which only includes strings that were made resources. If something is hardcoded it is in neither set, so the percentage is computed over the inventory rather than over the app — which is why this number is only trustworthy on a codebase where literals cannot reach the screen.

The drift case is the one I would build for first if I were doing it again, because it is the only part of this that is genuinely invisible without tooling. A locale falling from ninety-six to eighty-eight over three releases is not a thing anybody observes: no screen broke, no report arrived, and each individual feature added two or three keys that looked like nothing. A number computed per release makes that a visible line, and a visible line is what turns translation from a thing done in bursts after somebody complains into a thing kept up with.

That also reframes who the number is for. I built it imagining a release manager deciding whether to ship a language, which is the dramatic case and the rare one. The ordinary case is the person adding a feature, seeing that six locales just lost a little ground, and deciding whether that matters now or at the next translation pass — a small decision, made with information, dozens of times instead of once.

What this generalizes to#

A question that is being answered by impression is often computable, and the gap between those two is usually framing rather than difficulty.

The pattern is a quality that feels qualitative — is it complete, is it covered, is it consistent — sitting on top of two sets that both exist. Documentation against public API. Tests against modules. Schema fields against the records that use them. Accessibility labels against interactive elements. In each case somebody is giving a confident verbal answer, and in each case the comparison is a few lines.

So when you hear a percentage estimated in conversation, ask what it would take to compute. Often the answer is that both sides are already enumerable and nobody has written the subtraction — and once it exists, the interesting change is not that the number is known but that it is tracked, so the slow drift that nobody could see becomes a line that visibly moves.