What a small on-device model simply cannot do

Every small model has a ceiling. The uncomfortable property is not that it exists — it is that the model does not announce when it has hit one.

A large model and a small one both return fluent text. One of them is right more often, and nothing in the response says which situation you are in.

The case this post is about#

One assistant, three questions, on a device with no connectivity:

ask("summarise this note")        -> Local(…, confidence 0.91)
ask("what shipped this week")     -> Declined("requires current information")
ask(<40k-character document>)     -> Declined("exceeds local context")

The second and third rows are the subject. Without a Declined value they return a confident local answer — fluent, well-formed, the same latency and shape as a correct one — about information the model does not have.

Nothing downstream can tell those apart. A small model past its ceiling does not stall or error; it answers.

This post follows those three rows, because the ceiling has to be located deliberately and the return type has to have somewhere to put the answer we cannot do this.

The symptom#

The feature works in the demo. The demo inputs are short, well-formed and representative of what the person building it had in mind.

Real inputs are longer, messier, in a second language, contain a table, contain a contradiction, ask for something the model cannot know. The output stays fluent, because fluency is the thing small models are good at, and the failure is in the content rather than the form:

• it produces a confident answer that is wrong
• it follows the format but ignores an instruction further up the prompt
• it summarises the first part of a long input and silently drops the rest
• it loses a structure it was asked to maintain

None of these throws. All of them pass a test that asserts the output is non-empty and well-formed, which is why the gap between the feature runs and the feature works is so wide here.

Why the obvious fix failed#

The obvious fix is prompting. Be more specific, add examples, constrain the format.

Prompting genuinely moves the ceiling and it does not remove it, and the work has a sharply diminishing return that is easy to mistake for progress because each iteration improves the cases you are looking at. You can spend a very long time making the demo inputs better while the ceiling stays where it was.

The second attempt is to detect bad outputs and retry. Retrying the same model on the same input with the same ceiling is mostly a way to spend battery twice.

The actual mechanism#

Locate the ceiling deliberately, then design the behaviour at it.

Find the boundary by escalating difficulty on purpose. Build a set of inputs that get progressively harder along the dimension you care about — length, structure, reasoning depth, language — and find where output quality changes character. This is the same sweep discipline that locates a quantization cliff, applied to task difficulty instead of bit width.

Write the ceiling down as a product constraint. "Inputs under N words, single language, no nested structure" is a specification somebody can design to. An undocumented ceiling is rediscovered independently by every person who touches the feature.

Detect proximity to the boundary from the input, not the output. Input length, structure and language are observable before you spend the inference. A router that can decline up front is far cheaper and more reliable than one judging its own output afterwards.

Design the behaviour at the ceiling as a first-class state. Not an error. A state the interface supports: escalate to a larger model, ask the user to narrow the request, offer a non-AI path, or say plainly that this input is outside what the feature handles.

The fix#

The part most often skipped: build the escalation path before you need it.

If the only path is the local model, every ceiling encounter becomes a bad answer, because there is nowhere else to go. The system has no vocabulary for I should not answer this, so it answers.

This is the same argument as a verdict enum needing a value that is neither pass nor fail. A system

with only "answer" and "error" will answer.

The failure this prevents is worth being specific about, because it does not look like a failure. A small on-device model asked something past its ceiling does not stall, apologise, or return an error — it produces fluent, well-formed, confidently wrong text, at the same latency and in the same shape as a correct answer. Nothing downstream can distinguish the two. The absence of a declining value is therefore not a gap in error handling; it is a guarantee that the system will present its worst output with its full authority.

The escalation path has to exist before the ceiling is hit, which means it is a return type rather than a catch block:

sealed interface Answer {
    data class Local(val text: String, val confidence: Float) : Answer
    data class Escalated(val text: String, val via: String) : Answer
    data class Declined(val reason: String) : Answer      // the value that must exist
}

class Assistant(private val local: LocalModel, private val remote: RemoteModel?) {

    suspend fun ask(prompt: String): Answer {
        // Ceilings known IN ADVANCE, not discovered from a bad answer.
        val ceiling = when {
            prompt.length > local.contextChars   -> "exceeds local context"
            needsFreshFacts(prompt)              -> "requires current information"
            needsLongChain(prompt)               -> "multi-step reasoning beyond local capability"
            else                                 -> null
        }

        if (ceiling == null) {
            val r = local.complete(prompt)
            if (r.confidence >= MIN_CONFIDENCE) return Answer.Local(r.text, r.confidence)
        }

        // The escalation, and the honest floor when there is nowhere to escalate to.
        return remote?.let { Answer.Escalated(it.complete(prompt).text, via = "remote") }
            ?: Answer.Declined(ceiling ?: "local confidence below threshold")
    }
}

The three outcomes on a device with no connectivity:

ask("summarise this note")        -> Local(…, confidence 0.91)
ask("what shipped this week")     -> Declined("requires current information")
ask(<40k-character document>)     -> Declined("exceeds local context")

Declined is the whole design. Without it those last two return a confident local answer about information the model does not have, and nothing in the result says so.

Two details in that code earn their place and are easy to leave out.

The ceiling checks run before inference rather than after. A post-hoc confidence threshold catches the case where the model is unsure, and the expensive failures are the ones where it is confident and wrong — a question about this week's events has no uncertainty signal in it at all, because the model is answering fluently from a world that ended at its cutoff. Length and freshness are properties of the prompt, knowable without spending a single token, and that is the only place they can be checked reliably.

And Escalated carries via. A caller that cannot tell a local answer from a remote one cannot make any of the decisions that depend on the difference — whether the text may be cached, whether it left the device, whether it is billable, whether it is reproducible offline. One field, and it is the difference between a result and an audited result. whose only outcomes are answer and error will produce an answer in situations that deserve neither — and the value of a third outcome is that it tells whoever receives it to do something different rather than to doubt the answer they got.

Escalation is the obvious third outcome and not always the right one. Sending the input to a larger hosted model solves the capability problem by reintroducing every property the local model was chosen to avoid — the data leaves, the call costs money, and it fails when the network does. If the feature exists because processing stays local, then silently escalating is not a fallback, it is a reversal of the feature's premise that the user never agreed to.

Which makes the honest set of options smaller than it first appears: narrow the request, offer a non-AI path, escalate with the user's consent, or decline. All four are shippable. The one to avoid is the fifth, which is to answer anyway and hope — and that is the default the system falls into whenever nobody has built any of the other four.

How we proved it#

The boundary of this article, stated plainly.

Not measured: I have not benchmarked Gemini Nano, ML Kit or any specific small model against a difficulty ladder. I am not going to tell you where any particular model's ceiling is — that is a property of a model, a task and a prompt, and it moves with every version.

Measured, and the reason the structure above is what I would build: the value of an explicit third outcome, and the cost of not having one. The verdict vocabulary in this codebase includes a value meaning could not determine, which blocks a release exactly as a failure does while telling the reader to do something different — raise the budget, attach a device, look by hand. It earns its place regularly, and most often not because a judgement was close but because an evaluation could not be completed at all.

Also measured: that a check which cannot go red is not a check. An assertion written to verify something passed immediately because the words it searched for already appeared in the file for unrelated reasons. The lesson transfers directly: an output-quality check that has never rejected a real output is not validating anything, and the only way to know is to point it at something you know is bad.

What I expect to be true of the ceiling#

Three expectations, offered as things to check rather than findings.

Context length is the ceiling you will hit first. It is the most visible limit and the one most likely to fail silently — a model given more than it can attend to usually produces something, and that something is a summary of part of the input with no indication of the rest.

Instruction-following degrades before fluency does. A small model under pressure keeps sounding right while drifting from what it was asked, which is exactly the failure order that makes manual review unreliable.

The ceiling moves between versions, in both directions. A platform model updates outside your release cycle, so a constraint you documented can become wrong without anything in your code changing — which argues for the difficulty ladder being a suite you re-run, not an exercise you did once.

The second of those has a consequence worth stating. If fluency outlasts correctness, then the review that catches a bad answer cannot be a person skim-reading output — the output will read well. It has to be a check against something external: a known answer, a cross-check against a larger model, or a declared refusal that never got the chance to be wrong. Of those three, only the last costs nothing per request.

What this generalizes to#

When a component has a capability limit that it will not report, the design question is never how to push the limit — it is what the system does when it arrives there.

Find the boundary on purpose with a ladder of increasing difficulty. Write it down where product decisions get made. Detect proximity from the input rather than by judging the output. And give the system a third outcome, so that this is outside what I handle is a thing it can say — because a component whose only options are to answer or to fail will always answer.