When a local model beats the API, and the three numbers that decide it
The crossover is computable, the three numbers disagree with each other, and almost nobody computes any of them.
on-device
21 posts
What a local model does to your battery
Battery cost is attributed at app level and spent at feature level. Measure the delta against a control run, and treat thermal state as part of the result.
The first inference is the one users judge you on
Cold start is a different system from steady state, not the same one measured twice. Treat warmth as a resource with a lifecycle, and measure both numbers.
Core ML and LiteRT solve the same problem differently
A platform-native runtime and a portable one differ in ways the type system will not catch. What the shared declaration may promise is the whole design.
On-device semantic search over the user's own data
Semantic search on a phone is not a vector-database problem at user scale — it is an index-freshness problem, invisible until someone searches.
Designing the fallback for a device that cannot run your model
A fallback that silently became the default for 303 of 304 artifacts. Detect capability before attempting, and record on every artifact which path ran.
Gemini Nano on-device: what it actually costs you
On-device inference has no per-call bill, and that hides six real costs — eligibility, size, battery, thermal, capability ceiling and operations.
Routing between a local model and a remote one
Routing between a local and a hosted model needs a decision you can audit. Route before attempting, record the path on every result, and watch the proportion.
One inference API, two platform runtimes
Designing an expect/actual seam so platform differences stay on one side. What the shared declaration must not promise, and where the abstraction leaks.
How to know your on-device model got worse
A tiny eval suite that fails the build, the budget that bounds it, and why a verdict needs a third value meaning neither pass nor fail but undetermined.
Wiring ML Kit GenAI into a Compose screen without blocking it
Wiring an on-device GenAI capability into Compose is mostly state design. Model the availability and failure states before the happy path, not after.
Shipping a model in your APK without breaking the size budget
Bundling a model trades download size for a guarantee. The decision turns on first-run experience, and the budget needs a gate rather than good intentions.