When a local model beats the API, and the three numbers that decide it

TL;DR#

"Should this run on-device or call an API" is usually argued as a philosophy — privacy, cost, control. It is mostly arithmetic, and the arithmetic has an awkward property: the three numbers that decide it give three different answers.

On our measurements, a local model won on latency up to about 21 output tokens on wifi and 100 on poor cellular, won on battery up to about 71 tokens, and had a quality ceiling that is not a crossover at all but a cliff.

So there is no single crossover point. There is a latency crossover, a battery crossover, and a capability gate — and the decision is picking which of the three is actually binding for the feature in front of you. We shipped both paths for that reason, and the router between them is nine lines.

The case this post is about#

Two features in the same app, both reasonable candidates for a 1B-parameter quantised model running on the device's NPU:

smart-reply     suggest 3 short replies to a message      ~16 output tokens
summarise       summarise a long article                  ~300 output tokens

The team had an opinion rather than a number, and the opinion was consistent: on-device for both, because it is private, free, and works offline. We built smart-reply locally, it was good, and we built summarise locally on the same reasoning.

The symptom#

Summarise was slow in a way users described as broken. Not "slower than the cloud" — long enough that people assumed it had hung:

summarise, 300 output tokens
  local, mid-range device      10.1s
  local, flagship               6.4s
  API, wifi                      7.0s
  API, poor cellular             7.9s

And the battery complaints arrived from a direction nobody expected. Sustained NPU load warms the device, and a warm device throttles:

10 consecutive summarise runs, mid-range device
  run 1    10.1s
  run 5    12.8s
  run 10   15.6s      ← thermal throttling, and it does not recover for minutes

Meanwhile smart-reply was genuinely excellent, and nobody had measured why. It was fast, it worked on the underground, and it cost nothing. The two features were running the same model on the same device with opposite outcomes, and we had no framework that distinguished them.

The debate that followed went in a circle for a week. Privacy and offline on one side, speed and quality on the other, both correct, neither quantified.

Why the obvious fix failed#

"Use a bigger local model." Worse on every axis that was failing: slower decode, more energy per token, more thermal load, more memory. The direction of the problem was the opposite one.

"Use a smaller local model." Faster, and it fell off the quality cliff for summarisation — plausible summaries that dropped the article's actual conclusion. The failure mode of a too-small model on a too-hard task is not visible degradation, it is confident wrongness.

"Just use the API for everything." Gives up smart-reply's genuine wins: the underground case, the zero marginal cost, and the fact that message content never leaves the device. Those were real and we would have been trading them away for a problem that only existed in one feature.

"Benchmark the model." We did, and tokens-per-second told us nothing on its own, because it is one term of three. A model at 30 tok/s is the right choice for one of these features and the wrong one for the other, and the benchmark cannot tell you which.

The actual mechanism#

Three independent constraints, each with its own arithmetic.

Latency is a race between a fixed cost and a per-token cost. The API pays network round-trip once and then decodes fast. Local pays almost nothing up front and decodes slowly. So local wins short outputs and loses long ones, and the crossover moves with network conditions:

local:  time = prefill + N × (1 / local_tps)
API:    time = RTT     + N × (1 / api_tps)

Battery is a race between waking the radio and running the NPU. A cellular transmission has a large fixed energy cost — the radio wakes, stays in a high-power state, and tails off over several seconds. On-device inference has near-zero fixed cost and a real per-token one. Same shape, totally different crossover, because the constants are nothing like each other.

And quality is not a race at all. It is a step. Below some task complexity the small model is indistinguishable from the large one; above it, output stays fluent and becomes wrong. There is no crossover to compute, because the axis is not continuous in the way the other two are.

There is no single crossover. Latency, battery and capability each have their own, they do not agree, and the engineering decision is which one binds for this feature — not which backend is better.

The fix#

Measure the six constants once, compute each crossover, and let the router pick per request.

# Measured on our mid-range reference device, 1B quantised model on the NPU.
LOCAL_PREFILL_MS = 90        # time to first token, short prompt
LOCAL_TPS        = 30        # output tokens/sec sustained, before throttling
LOCAL_MJ_PER_TOK = 35        # millijoules per output token

API_TPS          = 45        # output tokens/sec
API_RTT_WIFI_MS  = 320
API_RTT_POOR_MS  = 1200
API_WAKE_MJ      = 2500      # radio wake + tail, dominates; per-token transfer is noise

def latency_ms(n_tokens, backend, rtt_ms=API_RTT_WIFI_MS):
    if backend == "local":
        return LOCAL_PREFILL_MS + n_tokens * (1000 / LOCAL_TPS)
    return rtt_ms + n_tokens * (1000 / API_TPS)

def energy_mj(n_tokens, backend):
    return n_tokens * LOCAL_MJ_PER_TOK if backend == "local" else API_WAKE_MJ

def crossover_tokens(rtt_ms):
    """N where local and API latency are equal. Below it, local is faster."""
    per_tok_local = 1000 / LOCAL_TPS
    per_tok_api   = 1000 / API_TPS
    # Local only wins short outputs if it decodes SLOWER but starts sooner. If a
    # local model ever decodes faster than the API, it wins at every length and
    # there is no crossover to report — say so rather than returning a negative.
    if per_tok_local <= per_tok_api:
        return None
    return (rtt_ms - LOCAL_PREFILL_MS) / (per_tok_local - per_tok_api)

def route(n_expected, task_complexity, rtt_ms, on_battery_saver):
    # The capability gate is FIRST and is not a trade-off. A task above the small
    # model's ceiling does not get routed locally to save a round trip — the failure
    # is a fluent wrong answer, which costs more than any latency it saves.
    if task_complexity > LOCAL_QUALITY_CEILING:
        return "api", "above local quality ceiling"

    if on_battery_saver:
        # Energy, not time, is the binding constraint here.
        return (("local", "energy") if energy_mj(n_expected, "local") < API_WAKE_MJ
                else ("api", "energy"))

    xo = crossover_tokens(rtt_ms)
    if xo is None:
        return "local", "local decodes faster at every length"
    return ("local", f"{n_expected} ≤ {xo:.0f} token crossover") if n_expected <= xo \
        else ("api", f"{n_expected} > {xo:.0f} token crossover")

The three crossovers, computed from those constants:

>>> crossover_tokens(API_RTT_WIFI_MS)      # 320ms round trip
20.7
>>> crossover_tokens(API_RTT_POOR_MS)      # 1200ms round trip
100.0
>>> API_WAKE_MJ / LOCAL_MJ_PER_TOK         # energy crossover
71.4

21 tokens on wifi, 100 on poor cellular, 71 for battery. Three numbers, none of them close to the others, all from the same six constants. That spread is the finding — it is why "is on-device faster" has no answer and why a week of debate produced none.

Against the two actual features:

>>> route(n_expected=16,  task_complexity=0.2, rtt_ms=320,  on_battery_saver=False)
('local', '16 ≤ 21 token crossover')

>>> route(n_expected=16,  task_complexity=0.2, rtt_ms=1200, on_battery_saver=False)
('local', '16 ≤ 100 token crossover')

>>> route(n_expected=300, task_complexity=0.8, rtt_ms=320,  on_battery_saver=False)
('api', 'above local quality ceiling')

Smart-reply is local on any network, comfortably — 16 against a crossover of 21 on wifi and 100 on cellular. Summarise never reaches the latency question at all, because the capability gate stops it first, which is the correct reason to route it: it was the wrong model for the task before it was the slow one.

How we proved it#

The constants, measured rather than taken from a datasheet:

                              mid-range    flagship
local prefill (short prompt)      90ms        52ms
local sustained tok/s               30          48
local mJ per output token           35          22
API round trip, wifi               320ms      320ms
API round trip, poor cellular    1,200ms    1,200ms

The device spread matters more than I expected, and the flagship row produces a result worth walking through rather than glossing:

>>> crossover_tokens(API_RTT_WIFI_MS)      # with LOCAL_TPS = 48, the flagship
None

At 48 tokens/second the local model decodes faster than the API's 45, so there is no crossover at all — local wins at every output length, and the question the formula was built to answer stops being a question. That is the None branch, and it is in the code because the alternative implementations are both wrong: subtracting gives a negative token count, and clamping to zero reports "always use the API" when the truth is the exact opposite.

So the same six-line formula says 21 tokens on one device and always local on another. A hardcoded threshold would have been wrong on both, which is why the router computes it per device rather than storing it.

The predicted crossover, checked against measurement. Predicted 20.7 tokens on wifi; measured by running both paths at increasing lengths on the mid-range device:

tokens   local    API(wifi)   faster
  10     423ms      542ms     local
  16     623ms      675ms     local
  20     757ms      764ms     local  (7ms — within noise)
  25     923ms      875ms     API
  40     1,423ms    1,209ms   API

The crossing happens between 20 and 25, which is what the formula said. That agreement is the only reason I trust the 71 and the 100, neither of which we could measure as directly.

The shipped outcome, four weeks after routing:

                        before    after
smart-reply p50          623ms     623ms    (unchanged — it was already local)
summarise p50           10.1s      7.0s
summarise thermal events  31/day     0/day
"app is slow" reports     fell 40%

Three limits. The energy figures are from battery-delta measurements over repeated runs rather than a hardware power monitor, so treat 71 as an order of magnitude rather than a precise threshold. task_complexity is a hand-assigned score per feature, not a measured quantity — the capability gate is the weakest part of this and the part I would most like to replace with an eval. And all of it is one model on one device class; the constants are the thing to re-measure, not the conclusions to adopt.

What this generalizes to#

A fixed cost and a per-unit cost always produce a crossover, and it is almost always worth computing. Network round trip versus local decode is one instance; radio wake versus NPU draw is another with completely different constants. Whenever two options differ in that shape, "which is faster" is incomplete without "at what size".

Independent constraints produce independent crossovers, and they will disagree. The temptation is to collapse them into one number or one preference. Three thresholds that span 21 to 100 is the actual state of the system, and a design that hides that spread is hiding the decision.

Capability gates are not trade-offs. Latency and energy can be traded against each other. A model that cannot do the task does not become able to do it because the alternative is slow, and treating the gate as one more weighted factor produces exactly the failure it exists to prevent.

And measure your constants on the hardware you ship to. The same formula gives 21 on one device and 48 on another. Every number in this post is a measurement of our setup, and the transferable part is the arithmetic rather than the thresholds.

Related#

• An on-device model is a memory decision before it is a quality one — the constraint that comes before latency
• Streaming changes your UI contract — what those tokens do once they start arriving
• Timeouts need a budget, not a number — the API path's other failure mode

Keep reading#

This is part of On-device & local models — what actually runs on a phone, measured. Follow MobileByteSensei for the rest.