The on-device privacy argument, stated honestly

"It runs on your device" is a true statement about where a matrix multiplication happens.

It is not, by itself, a statement about whether anything concerning the user leaves the device — and those two get collapsed into one marketing line constantly, including by people who know better and are not trying to mislead anyone.

The case this post is about#

One telemetry record, and the field that kept being proposed for it:

refused: telemetry carries non-measurement field(s): promptText
refused: telemetry carries non-measurement field(s): firstWords
{"event":"inference","modelVersion":"nano-2026.08","promptChars":412,"outputChars":180,
 "durationMs":214,"accepted":true,"errorClass":null,"promptHash":"9f2c1b05e4a7"}

The second refusal is the one the design exists for. firstWords is not the prompt — it is four words of it, proposed in good faith, with the reasonable argument that four words cannot identify anybody.

It is also content, and a reviewer comparing it against a policy document would have had to make a judgement. This post follows that one field, because "the model runs on the device" and "nothing about the user leaves the device" are different claims, and only the second one is a privacy property.

The symptom#

The architecture is genuinely local. The weights ship in the app, inference runs on the NPU, no request goes to an inference endpoint. Everything the diagram claims is accurate.

Then the ordinary machinery of shipping software does its job:

• a crash report attaches the state that caused it
• an analytics event records which suggestion the user accepted
• a support bundle includes the log that includes the prompt
• a quality metric reports the output's length, and later its content
• a sync feature backs up the user's notes, which now contain generated text

None of those is a betrayal. Each was added by somebody solving a real problem, and each moves something about the user off the device. The inference stayed local; the data did not, and no single decision was the wrong one.

Why the obvious fix failed#

The obvious fix is a policy: we don't send user content off the device. Written down, agreed, in the handbook.

A policy is a statement about intentions held by people who are not in the room when the code is written. It fails the way all unenforced guidance fails — not through defiance, but because somebody adds a debug log during an incident, ships it, and the line survives into production.

I can be specific about this failure mode because I shipped it. The guidance in this codebase said: never print a secret into the transcript, and it named the obvious offenders. On 2026-09-03 a command ran that was not on the list, against a file that was malformed in a way nobody anticipated, and a credential went straight into the output. The policy was clear, agreed, and present. It did not refuse anything, because a policy cannot — only a mechanism refuses.

The actual mechanism#

What makes a boundary real is that something stops at it, and that you can demonstrate the stopping without trusting anyone's account of it.

Three properties, in the order they matter:

Default-deny, with a decidable allowlist. The boundary refuses everything by default, and what passes is a list whose membership is settled by a single checkable question rather than by judgement. In the secrets case that question is can this operation emit file content? For a user-data boundary it is does this payload contain anything derived from user input? A list you can decide is a list you can audit; a list of exceptions people argued for is not.

Metadata instead of content. This is the part that sounds like a compromise and is not. Verifying a credential materialized correctly never required reading it — the file exists, it is 3,247 bytes, its hash changed. Every check that looked like it needed the value turned out to have a metadata-shaped answer.

The same holds for user data. You can know a feature ran, how long it took, how many tokens it produced, whether the user accepted it, and whether it failed — without ever recording what it said. If a check seems to require the content, that is a signal the check is wrong, not that the boundary needs an exception.

A disclosed failure path. Boundaries get crossed. What distinguishes a real one is what happens next: name exactly what was exposed, treat the channel it went through as compromised, rotate what can be rotated, and say so. A boundary with no stated breach procedure has not been thought about past the diagram.

The fix#

The practical rule I would give anyone shipping a local model: log the measurement, never the material.

Counts, durations, hashes, acceptance rates, error classes, model version. All of that is enough to run a feature responsibly, debug it, and tell whether a release made it worse. None of it is the user's sentence.

The temptation to keep just one content field is strong and always arrives wearing a good reason — we cannot debug this without seeing the input. Sometimes that is true, and the honest response is to build a path where the user chooses to send it for that incident, not to widen the default so the debugging is convenient later.

The way to make that hold under pressure is a shape with no slot for content, enforced on the way out rather than agreed in review:

// Every field here is a MEASUREMENT. There is deliberately nowhere to put a sentence.
type Telemetry = {
  event: string;
  modelVersion: string;
  promptChars: number;        // the LENGTH, not the prompt
  outputChars: number;
  durationMs: number;
  accepted: boolean;
  errorClass: string | null;
  promptHash: string;         // stable across runs, reveals nothing
};

const ALLOWED = new Set<keyof Telemetry>([
  "event", "modelVersion", "promptChars", "outputChars",
  "durationMs", "accepted", "errorClass", "promptHash",
]);

function emit(record: Record<string, unknown>): string {
  // An extra field is a HARD error, not a stripped one. Stripping is silent, and a
  // silent strip means the next person adds a field and believes it is being sent.
  const extra = Object.keys(record).filter((k) => !ALLOWED.has(k as keyof Telemetry));
  if (extra.length) {
    throw new Error(`telemetry carries non-measurement field(s): ${extra.join(", ")}`);
  }
  // Belt and braces: a measurement field that holds prose is still prose.
  for (const [k, v] of Object.entries(record)) {
    if (typeof v === "string" && /\s/.test(v) && k !== "event" && k !== "errorClass") {
      throw new Error(`field "${k}" contains whitespace — looks like content, not a measurement`);
    }
  }
  return JSON.stringify(record);
}

The field that kept being proposed, and the record that shipped:

const base = { event: "inference", modelVersion: "nano-2026.08", promptChars: 412,
               outputChars: 180, durationMs: 214, accepted: true, errorClass: null,
               promptHash: "9f2c1b05e4a7" };
try { console.log(emit({ ...base, promptText: "summarise my medical notes from Tuesday" })); }
catch (e) { console.log(`refused: ${(e as Error).message}`); }
try { console.log(emit({ ...base, firstWords: "summarise my medical" })); }
catch (e) { console.log(`refused: ${(e as Error).message}`); }
console.log(emit(base));

refused: telemetry carries non-measurement field(s): promptText
refused: telemetry carries non-measurement field(s): firstWords
{"event":"inference","modelVersion":"nano-2026.08","promptChars":412,"outputChars":180,
 "durationMs":214,"accepted":true,"errorClass":null,"promptHash":"9f2c1b05e4a7"}

The second refusal is the one the design is for. firstWords is not the prompt, it is four words of it, and it arrived with the reasonable argument that four words cannot identify anybody. It is also content, it was proposed by somebody acting in good faith, and a reviewer comparing it against a policy document would have had to make a judgement. A type with no slot for it does not require one.

How we proved it#

The measured part is the inversion, and the fact it was forced by a real leak rather than designed in advance.

before   policy: "never print a secret"        enforced by: nobody
after    default-deny on secret paths          enforced by: a hook, at call time
check    does the operation emit content?      decidable per candidate

The thing worth taking from it is not the hook. It is that the policy and the mechanism had been describing the same intention for months, and only one of them ever stopped anything. Everybody agreed with the policy, including the command that violated it.

The honest limitation: the mechanism protects the channel it is attached to. It knows about commands run in one place. It has nothing to say about a library that phones home, an SDK that collects more than its documentation admits, or a backup that syncs a directory somebody later started writing generated content into. Each of those needs its own boundary, and the existence of one good one creates a feeling of coverage it has not earned.

The tension I cannot resolve#

Being straight about the part I do not have an answer to.

The moment you want to know whether your on-device feature is working — not on your phone, across everyone's — you need information from devices you do not hold. That is telemetry, and telemetry is egress. The privacy property and the observability need are in direct opposition, and the opposition is structural rather than a design failure.

I hit this in a different context and wrote about it: a fallback path had silently become the default for 303 of 304 artifacts, and the only reason that was discoverable was a field recorded on every artifact that a grep could count. On one machine that census is a local file read. Across a fleet it is a reporting pipeline, and every question about what you are allowed to collect comes back.

The partial answer is that metadata-only telemetry is genuinely weaker evidence but still useful — you can learn that acceptance dropped 12% without learning what anybody wrote. The part I will not claim is that this is free. You see less. A content-free census answers something changed and not what changed, and sometimes that is the difference between finding a regression and knowing one exists.

What this generalizes to#

Separate the two claims before you make either. The model runs on your device is about compute. Nothing about you leaves is about every path out, most of which have nothing to do with the model — and only the second one is what a user hears.

Then make the second claim true the only way a claim like that becomes true: a boundary that refuses by default, an allowlist whose membership is decidable, measurements rather than material in everything you record, and a stated procedure for the day it is crossed anyway. If what you have is a paragraph in a handbook, you have an intention. Mine was clear, agreed, and did not refuse a thing.