The first inference is the one users judge you on

Every benchmark you will read quotes the warm number. Every user's first impression is the cold one.

Those are not the same system measured twice. They are different systems, and only one of them has been optimised — usually the one nobody meets first.

The case this post is about#

One on-device model, across five requests, with a 60-second idle timeout:

inference 1   cold   1,840ms   (session created)
inference 2   warm      210ms
inference 3   warm      195ms
   ... 95s idle, timeout is 60s ...
inference 4   cold   1,790ms   (evicted, session recreated)
inference 5   warm      205ms

"About 200ms" is true of three of those five. Two of them took nine times that, and both were the first thing a user did after picking the phone up.

This post follows those five numbers, because the steady-state figure is the one that gets quoted and the one nobody experiences first — and because inference 4 is an outlier with no visible cause unless something is counting evictions.

The symptom#

The model loads on first use. Weights are read from storage, memory is allocated, a delegate or accelerator is initialised, maybe a graph is compiled for this specific device.

None of that happens on the second call. So you end up with two latencies sharing one name, and every artefact in the project quotes the smaller one — the benchmark, the design doc, the ticket that says inference is fast enough. The first call can be an order of magnitude slower and nothing in the codebase says so, because the number was measured after warm-up without anyone deciding to exclude it.

Then it gets worse in a way that is easy to miss: the process dies. The user switches apps, the system reclaims memory, and the next interaction is a cold start again. Cold start is not a one-time cost paid at install. It is a recurring cost whose frequency is set by memory pressure you do not control.

Why the obvious fix failed#

The obvious fix is to warm up at launch — load the model when the app starts, so it is ready when the user arrives.

It moves the cost rather than removing it, and often to a worse place. Launch is already the most contended moment in an app's life, competing with layout, network calls and whatever else wants the first second. Loading a model there can make the whole app feel slow in exchange for making one feature feel fast.

The second attempt is to warm up lazily on the screen before the one that needs it. Better, and it depends on a navigation assumption that stops being true the moment someone adds a deep link.

The actual mechanism#

Treat warmth as state you own, with an explicit lifecycle, rather than a side effect of having called something recently.

Warm on intent, not on launch or on arrival. The signal is the user doing something that implies they are heading toward the feature — opening the section, focusing the input, beginning to type. That is early enough to hide most of the cost and late enough to avoid paying it for users who never go there.

Know which state you are in, and make the UI say so. A cold call and a warm call need different affordances. The warm one can be instant with no indicator at all; the cold one needs to admit it is doing something, and admitting it is far better than a frozen control.

Protect warmth deliberately. Once warm, the expensive thing stays warm until there is a reason to release it — and "the screen was closed" is usually not a good enough reason if the user is likely to come back within seconds.

The fix#

The principle I am most confident about transfers from a different domain, and I will be explicit that this is where my evidence actually comes from.

In this codebase there is a hard rule against killing the build daemon. Not a preference — a mechanical guard that blocks the commands. The reason is that a warm daemon with its caches gives near-instant incremental builds, and killing it forces a cold reconfigure-and-recompile measured in tens of minutes. The warm and cold paths differ by orders of magnitude, and the cold path is reached by a single careless command.

What made that manageable was not making the cold path faster. It was:

• recognising warmth as a resource with a lifecycle rather than an accident
• making the thing that destroys it hard to do by accident
• never quoting the warm number as though it were the only one

All three transfer directly to a model held in memory. The analogy is not decorative: in both cases an expensive initialisation produces a reusable warm state, the warm state is invisible in the code, and the cost of losing it is paid by whoever shows up next.

Treating warmth as a resource means it has a lifecycle something owns, and the honest version reports both numbers rather than the one that flatters:

class WarmSession<T>(
    private val initialise: () -> T,
    private val idleTimeoutMs: Long,
    private val clock: () -> Long,
) {
    private var held: T? = null
    private var lastUsedAt = 0L
    val stats = Stats()

    data class Stats(var cold: Int = 0, var warm: Int = 0, var evicted: Int = 0,
                     var coldMsTotal: Long = 0, var warmMsTotal: Long = 0) {
        // Quoting only the warm number is the thing the rule forbids.
        fun report() = "cold ${cold}x avg ${if (cold > 0) coldMsTotal / cold else 0}ms · " +
            "warm ${warm}x avg ${if (warm > 0) warmMsTotal / warm else 0}ms · evicted ${evicted}x · " +
            "p-cold ${"%.0f".format(cold * 100.0 / (cold + warm))}%"
    }

    fun <R> use(block: (T) -> R): R {
        val start = clock()
        // Eviction is the thing that destroys warmth, and it has to be VISIBLE —
        // a silent eviction makes the next request look like an inexplicably slow one.
        if (held != null && start - lastUsedAt > idleTimeoutMs) {
            held = null
            stats.evicted++
        }

        val wasCold = held == null
        val session = held ?: initialise().also { held = it }

        val elapsed = clock() - start
        if (wasCold) { stats.cold++; stats.coldMsTotal += elapsed }
        else         { stats.warm++; stats.warmMsTotal += elapsed }

        lastUsedAt = clock()
        return block(session)
    }
}

A realistic access pattern — a burst, a gap longer than the timeout, another burst:

inference 1   cold   1,840ms   (session created)
inference 2   warm      210ms
inference 3   warm      195ms
   ... 95s idle, timeout is 60s ...
inference 4   cold   1,790ms   (evicted, session recreated)
inference 5   warm      205ms

cold 2x avg 1815ms · warm 3x avg 203ms · evicted 1x · p-cold 40%

The last line is the number worth quoting and the one nobody does. "About 200ms" is true of three of those five calls; 40% of them took nine times that, and both of the slow ones were the first thing a user did after picking the phone up. The eviction counter is what connects the two — without it, inference 4 is an outlier with no cause, and the obvious next move is to go looking for one in the inference code.

How we proved it#

Being precise, because this is the article where the boundary matters most.

Measured: the warm-versus-cold asymmetry in build tooling, where it is extreme and well understood, and where the response was a mechanical guard rather than guidance. Also measured: that the guard was necessary — the rule exists because the cold path kept getting triggered by hand.

Not measured: I have not profiled cold versus warm inference for Gemini Nano, ML Kit or Core ML on a device. I do not have a ratio to quote you, and I am deliberately not quoting one, because the only honest numbers here are the ones you measure on your own device and model.

measured      warm vs cold in build tooling      extreme, mechanically guarded
not measured  warm vs cold for on-device inference

What I would do first, and what I would recommend before any of the design above: measure your own first-call and tenth-call latency separately, on a mid-range device, and write both numbers down where the team can see them. Almost every decision in this article follows from the size of that gap, and the gap is the one thing nobody can tell you.

What I expect to be different for inference#

Three places the build-tooling analogy will not carry, stated as expectations.

You do not control eviction. A build daemon lives until something kills it. A model in a mobile process is subject to the system reclaiming memory on its own schedule, so warmth is not just something you preserve — it is something that can be taken, and the code has to handle discovering that mid-session.

Warm is not binary. Weights loaded, accelerator initialised, graph compiled, caches populated: these are separate costs that can be in different states. A "warm" model may still pay a real cost on a first call with a new input shape.

Thermal state interferes. Holding a model warm is not free, and a device that is already hot may behave differently from one that is not — which means a warm call is not even reliably one number.

That last one is why I would measure a distribution rather than a value, and why a single impressive benchmark figure is the least useful artefact you can produce here.

What this generalizes to#

Where a first operation is much more expensive than subsequent ones, you have two systems and one name, and the name almost always refers to the cheap one.

Measure both and record both. Warm on a signal of intent rather than on launch or arrival. Let the interface tell the truth about which path it is on. Protect the warm state as a resource, and make destroying it require intent — because the thing about warmth is that nobody notices it until it is gone, and by then the person experiencing the cold path is a user rather than you.