Gemini Nano on-device: what it actually costs you

The pitch for on-device inference is that it is free. No per-token billing, no API key, no spend to forecast.

That is true about money and it is the only axis on which it is true. The costs are real; they are just denominated in things that do not arrive as an invoice, which makes them easy to leave out of the decision that commits you to them.

The case this post is about#

One decision — ship a summarisation feature on-device — and the six-row table that was supposed to justify it:

{
  decidable: false,
  blockers: [
    'unowned: operations',
    'unpriced — spike these first: battery, thermal, operations'
  ],
  irreversible: [ 'eligibility' ]
}

One row had a number and it was zero: no per-call cost. That is the row the decision was being celebrated on.

Three rows had no number at all, one had no owner, and the only irreversible row — 62 per cent of the install base getting the feature and 38 per cent not — was fully priced and had been read as a capability statement rather than a product commitment.

This post follows those six rows, because the decision was not wrong on the axis it was argued on. It was not yet decidable on any of the others.

The symptom#

The decision is made on the strongest argument — no marginal cost, data stays local — and it is a good argument. What makes it risky is that the comparison is usually run against the one cost an API has that is easy to quantify, so everything hard to quantify is implicitly valued at zero.

Then the costs arrive, individually, owned by people who were not in the conversation:

• a device-eligibility matrix that product has to design around
• a size increase that nobody budgeted
• a battery complaint that support receives
• a thermal behaviour that QA cannot reproduce
• a capability ceiling that is discovered by users
• an operational surface with no status page and no vendor

Why the obvious fix failed#

The obvious fix is to prototype it. Build the feature, run it, see if it is good.

A prototype answers can this work and the costs above are almost all answers to what happens at scale, on devices I do not have, over time. A prototype on a recent phone, plugged in, with a developer's inputs, is the single most favourable measurement the feature will ever produce — and it is the measurement the decision gets made on.

The second attempt is to add a spike for the unknowns. Better, and spikes tend to be scoped to the technical questions (does it run, how fast) rather than the organisational ones (which devices do we drop, who owns the battery complaint).

The actual mechanism#

Enumerate the costs before building, and assign each one a number or an owner. Six categories, and the first is the one that reshapes the product.

Device eligibility. On-device models require hardware that not all of your install base has. That produces a fork in the product: a feature some users have, which means a non-AI path, a different onboarding, and a support story. This is a product decision, not a technical one, and it should be made before the code exists.

Distribution size. The weights live somewhere, and every delivery option costs something — binary size, a first-run download, or a platform mechanism with its own constraints.

Battery. Sustained compute, often with the screen on. Attributed to your app at a granularity that cannot be broken down per feature, reported to the user with your icon beside it.

Thermal. A warm device throttles, so the feature gets slower exactly when it has been used most, which reads as the app degrading.

Capability ceiling. A small model has a hard limit it will not report. The cost here is the escalation path, the detection, and the product behaviour at the boundary — all of which is work that does not exist in the hosted-model version.

Operations. No status page, no vendor to escalate to, no server-side fix. A bad model behaviour is shipped into installs you do not control, and the rollback is a release.

The fix#

The practical move: write the six down as a table, with a number or an owner in every row, before committing. Rows you cannot fill are the ones to spike.

What makes this worth the ceremony is that each cost lands on a different team, and none of them is in the room when "no per-call cost" is being celebrated. Size belongs to whoever owns the install funnel. Battery and thermal belong to QA and support. Eligibility belongs to product. Operations belongs to whoever is on call.

A cost with no owner is not a cost anybody is tracking, and the characteristic failure of the on-device decision is not that it was wrong — it is that it was made against one axis and paid on six.

The table earns the ceremony by refusing to total up until every row has a number and an owner, which is what stops it becoming a document:

type Row = {
  cost: string;
  owner: string | null;
  value: number | null;
  unit: string;
  reversible: boolean;
};

function appraise(rows: Row[]) {
  const unowned = rows.filter((r) => r.owner === null).map((r) => r.cost);
  const unpriced = rows.filter((r) => r.value === null).map((r) => r.cost);
  const irreversible = rows.filter((r) => !r.reversible).map((r) => r.cost);

  const blockers: string[] = [];
  // A row with no owner is not being tracked by anyone, which is the actual
  // failure mode — not a wrong number, an unassigned one.
  if (unowned.length) blockers.push(`unowned: ${unowned.join(", ")}`);
  if (unpriced.length) blockers.push(`unpriced — spike these first: ${unpriced.join(", ")}`);

  return {
    decidable: blockers.length === 0,
    blockers,
    // Surfaced separately because it is the only row that cannot be worked on
    // after shipping, so it belongs in the decision rather than the backlog.
    irreversible,
  };
}

The six rows as they stood the day the decision was being celebrated for having no per-call cost:

console.log(appraise([
  { cost: "per-call",    owner: "finance",  value: 0,    unit: "USD/call",    reversible: true },
  { cost: "apk-size",    owner: "growth",   value: 31,   unit: "MB",          reversible: true },
  { cost: "battery",     owner: "qa",       value: null, unit: "mAh/100 inf", reversible: true },
  { cost: "thermal",     owner: "qa",       value: null, unit: "degC rise",   reversible: true },
  { cost: "eligibility", owner: "product",  value: 62,   unit: "% of base",   reversible: false },
  { cost: "operations",  owner: null,       value: null, unit: "days to fix", reversible: true },
]));

{
  decidable: false,
  blockers: [
    'unowned: operations',
    'unpriced — spike these first: battery, thermal, operations'
  ],
  irreversible: [ 'eligibility' ]
}

One row had the number everybody was looking at, and it was zero. Three had no number, one had no owner at all, and the only irreversible row — 62% of the install base getting the feature and 38% not — was fully priced and nobody had read it as a product commitment. That is the whole appraisal: the decision was not wrong on the axis it was argued on, and it was not decidable yet on any of the others.

The eligibility row deserves singling out, because it is the only one that can make the decision irreversible. Size, battery and thermal are all costs you can work on after shipping. A device eligibility fork is a product commitment: once some users have the feature and some do not, you have two onboarding paths, two support stories and a segment of your base whose experience you have to keep designing for separately. Walking that back later means removing a feature from the people who have it, which nobody ever does.

Operations is the row most often valued at zero, and it is worth pricing honestly. With a hosted model, a bad behaviour is a server-side change — minutes, no release. On-device, the same fix is a store review, a staged rollout, and a population of installs that will keep running the old behaviour for as long as they keep not updating. The engineering cost is similar; the time to remediate is different by orders of magnitude, and that difference is the one you will feel at the worst possible moment.

How we proved it#

Not measured: I have not shipped Gemini Nano. I have no device-eligibility figures, no weight sizes, no battery measurements, no latency numbers and no capability benchmarks, and I am not supplying any — those are specific to a model version, a device and a task, and a number invented here would be more harmful than the absence of one.

Measured, and the reason for the table: the pattern where a question with no owner degrades silently. A concrete one from this week: a build step here ran an SEO audit against the site's output directory and reported all 113 posts declare og:image — green, specific, confident. The output directory was two days stale, the content directory held 145 posts, and the audit had just certified a build that did not contain any of the recent work. Nothing failed. The check was present, it ran, and it described the wrong artifact, because is the thing I am auditing current was a question nobody owned.

The transfer is direct: every row in the table above is a question somebody has to own, and the default outcome for an unowned one is that it degrades silently until a user reports it.

What I would want to know first#

If I were making this decision tomorrow, in order:

What fraction of our install base is eligible? This is usually knowable before any code is written and it determines whether you are building one feature or two.

What does a realistic session cost in battery, measured as a delta against the same session without the feature? The absolute number means nothing; the difference is the whole answer.

Where is the capability ceiling for our actual task? Not in general — for the inputs our users send, which is a difficulty ladder somebody has to build.

Those three reshape the design. Everything else is tuning.

What this generalizes to#

When something has no marginal cost, the comparison against the thing that does will be made on price and won on price, and every cost that is not denominated in money gets implicitly valued at zero.

List them anyway. Give each a number or a name of someone who owns it. The costs that go unlisted do not go unpaid — they get paid later, by whoever happens to be holding that part of the system, who will have no context for why it suddenly became their problem.