When a local model beats the API, and the three numbers that decide it
The crossover is computable, the three numbers disagree with each other, and almost nobody computes any of them.
inference
10 posts
What a local model does to your battery
Battery cost is attributed at app level and spent at feature level. Measure the delta against a control run, and treat thermal state as part of the result.
The first inference is the one users judge you on
Cold start is a different system from steady state, not the same one measured twice. Treat warmth as a resource with a lifecycle, and measure both numbers.
Core ML and LiteRT solve the same problem differently
A platform-native runtime and a portable one differ in ways the type system will not catch. What the shared declaration may promise is the whole design.
Designing the fallback for a device that cannot run your model
A fallback that silently became the default for 303 of 304 artifacts. Detect capability before attempting, and record on every artifact which path ran.
Routing between a local model and a remote one
Routing between a local and a hosted model needs a decision you can audit. Route before attempting, record the path on every result, and watch the proportion.
One inference API, two platform runtimes
Designing an expect/actual seam so platform differences stay on one side. What the shared declaration must not promise, and where the abstraction leaks.
How to know your on-device model got worse
A tiny eval suite that fails the build, the budget that bounds it, and why a verdict needs a third value meaning neither pass nor fail but undetermined.
Quantization has a quality cliff, and you can see it
Quantization degrades non-linearly, so an average score hides the cliff. Find it with a suite that fails per-case, and keep the per-case results, not the mean.
What a small on-device model simply cannot do
A small local model has a hard ceiling, and the design question is what happens at it. Find the boundary deliberately, then build the escalation path first.