Streaming a response changes your UI contract, not just your latency

TL;DR#

Switching a model response from a single result to a token stream looks like a latency improvement with no other consequences. It changes the UI's contract: instead of rendering one final value, you are now rendering every prefix of that value, and some prefixes are wrong in ways the final answer is not.

"The dosage is 500" is a valid prefix of "The dosage is 500mg for adults, but consult a doctor". Shown for 400ms, it is a different and worse statement. Markdown mid-render, a truncated code block, a half-written number, a refusal that begins with a compliant-sounding clause — all are states a non-streaming UI never had.

Decide what you render at each prefix, deliberately. Streaming is a product decision, not a transport optimisation.

The case this post is about#

A medical-reference assistant, moved from blocking to streaming because first-token latency was 4.2s:

// before
suspend fun ask(q: String): String = api.complete(q).text

// after
fun ask(q: String): Flow<String> = api.stream(q)        // cumulative text per chunk

// the UI, changed in the obvious way
val text by vm.answer.collectAsStateWithLifecycle()
MarkdownText(text)       // re-renders on every chunk

First token dropped to 310ms. Everything else about the change was treated as a rendering detail.

The symptom#

A safety problem, reported by a clinician user:

t=0.3s   "The maximum dose is 4000"
t=0.7s   "The maximum dose is 4000mg per day"
t=1.4s   "The maximum dose is 4000mg per day for adults under 65 with no hepatic impairment"

For 400ms the screen said 4000, with no unit. For another 700ms it said 4000mg per day with no qualifying conditions — which is a clinically different and unsafe statement. The final answer was correct and appropriately hedged; three of its prefixes were not.

A refusal that read as compliance:

t=0.4s   "You can take 20 tablets"
t=1.1s   "You can take 20 tablets only if— I can't advise on that. Please consult…"

The model's refusal began by restating the premise. Mid-stream, it looked like an answer.

Markdown rendering mid-token:

t=1.2s   "Dosage table:\n| Drug | Dose |\n|---|"      → a broken table, then a flash of raw pipes
t=1.3s   code fence opened, not closed                 → the rest of the answer rendered as code

And a copy button that copied a prefix. A user tapped Copy at 1.1s and pasted "The maximum dose is 4000mg per day" into a note. The qualifier was never in their clipboard.

Why the obvious fix failed#

"Debounce the rendering." We rendered every 300ms instead of every chunk. It reduced flicker and the 400ms unit-less number still appeared — a debounce changes how often you show a prefix, not which prefixes are safe.

"Only render complete sentences." Better, and it broke on the clinically important case: "The maximum dose is 4000mg per day." is a complete sentence and an incomplete statement. Sentence boundaries are not semantic boundaries.

"Stream but hide until done." Which is the blocking behaviour with extra machinery. It is sometimes the right answer — and if it is, you did not need streaming.

"Add a disclaimer under the streaming text." A static "answer in progress" label under a confidently-wrong number. Users read the number.

The actual mechanism#

A blocking call has one observable state. A stream has N, and your UI renders all of them.

blocking:   invisible → final                       1 user-visible state
streaming:  prefix₁ → prefix₂ → … → prefixₙ         n user-visible states

Three properties follow:

Every prefix is a claim. Text on screen is read as a statement, regardless of your intent that it is provisional. "4000" for 400ms is a thing the app said.

Prefixes are not a subset of valid outputs. This is the key point. A model's output distribution is over complete responses; the prefixes of a safe response include unsafe statements. Validating the final answer says nothing about its prefixes.

And syntactic structures span tokens. Markdown tables, code fences, JSON, LaTeX, links — all have an opening and closing token, so every intermediate state is syntactically invalid. A renderer fed a prefix produces garbage or, worse, plausible garbage.

The second property is the one that breaks the usual mitigation, so it is worth stating as starkly as it deserves: a safety check on the completed response is not a check on what the user read. They read the prefixes. By the time the validator sees a complete answer and rejects it, every intermediate state has already been displayed — and retracting them is a different product problem than preventing them.

t=0.4s   "The maximum dose is 4000"              ← displayed, and wrong without its unit
t=0.9s   "The maximum dose is 4000 mg per day"   ← displayed, still missing the qualifier
t=1.6s   "The maximum dose is 4000 mg per day
          for healthy adults under 65 without
          liver impairment — consult a clinician" ← the sentence the validator sees

Only the last line was ever validated. The first two were on screen for over a second each, are the ones a screenshot captures, and are what a reader skimming has already acted on. Each is a truncation of a correct answer, which is precisely why no amount of improving the model's final output addresses them.

Streaming makes every prefix a product surface. The question is not "how do we render faster" but "which partial states are we willing to show", and for some content the answer is none.

The fix#

Choose a render unit with semantic meaning, and gate the interactions that capture state.

1. Stream structure, not raw text. Have the model emit segments, so the UI knows what is complete:

// the model is instructed to emit one JSON object per logical unit
// {"kind":"answer","text":"..."}  {"kind":"caveat","text":"..."}  {"kind":"done"}
fun ask(q: String): Flow<Segment> = api.stream(q)
    .let { parseNdjsonSegments(it) }       // only emits COMPLETE segments

sealed interface Segment {
    data class Answer(val text: String) : Segment
    data class Caveat(val text: String) : Segment
    data object Done : Segment
}

A segment is a unit the author of the prompt declared meaningful. Rendering per segment means never showing half a claim.

2. Hold the answer until its caveats have arrived, which is the safety fix:

// commonMain
val display: StateFlow<AnswerState> = segments
    .scan(AnswerState.Loading) { state, seg ->
        when (seg) {
            is Segment.Answer -> state.copy(answer = seg.text, complete = false)
            is Segment.Caveat -> state.copy(caveats = state.caveats + seg.text)
            Segment.Done      -> state.copy(complete = true)
        }
    }
    .stateIn(scope, WhileSubscribed(), AnswerState.Loading)

@Composable
fun AnswerView(state: AnswerState) {
    when {
        state.answer == null   -> ThinkingIndicator()           // streaming, nothing shown
        !state.complete        -> AnswerWithPendingNotice(state) // answer + "still loading caveats"
        else                   -> FullAnswer(state)
    }
}

For this product we went further: in the medical context the answer is not shown at all until complete, with a progress indicator instead. Streaming still improves perceived latency, because the indicator is immediate and meaningful — and no unqualified dose figure is ever on screen.

3. Gate the capture interactions on completion:

IconButton(onClick = { clipboard.setText(state.fullText()) },
           enabled = state.complete) { CopyIcon() }       // ← no copying a prefix

Copy, share, save, "use this answer" — every one of them captures the current state, and every one must wait.

4. And render structured content only when its structure closes:

// a markdown block is rendered only once its fence/table is complete
fun completeBlocks(markdown: String): List<String> =
    markdown.splitIntoBlocks().dropLastWhile { it.hasUnclosedFence() || it.isPartialTable() }

For a different product, a different answer. A code assistant streaming into an editor should show tokens as they arrive — partial code is useful, the user is watching it form, and nothing is being asserted as fact. The decision is per content type:

content type          safe to stream raw?   why
prose explanation     yes                   prefixes are partial, not false
code                  yes                   the user reads it as in-progress
a number with a unit  NO                    the prefix is a different quantity
a hedged claim        NO                    the hedge may arrive last
a refusal             NO                    the opening clause may restate the premise
structured markup     NO                    invalid until it closes

How we proved it#

That unsafe prefixes existed, enumerated rather than assumed. We replayed 400 logged responses and checked every prefix against a safety rubric:

400 final answers, all passing the safety review
  responses with ≥1 prefix that failed the same review:   112   (28%)
  of those, prefixes shown for >250ms:                     89
  worst case: an unqualified dose figure shown for 740ms

Twenty-eight percent of safe answers had an unsafe prefix. That number is the reason the product hides the answer until complete, and it is not something any amount of final-output validation would have surfaced.

That the fix removes them:

same 400 responses, segment-gated rendering
  prefixes failing the safety review that reach the screen:   0

Perceived latency, which is what streaming was for:

                           blocking   raw streaming   segment-gated
time to first feedback        4,200ms       310ms          340ms
time to a complete answer     4,200ms     2,900ms        3,050ms
user-rated responsiveness
  (5-point, n=42)                2.4          4.1            3.9

Segment-gating keeps almost all the perceived-latency win. 3.9 against 4.1 is the cost of not showing unqualified text, and it is a price the product owner chose knowingly.

And the copy-a-prefix case:

copy taps during streaming, 2 weeks
  before:  1,840 taps, 312 (17%) captured an incomplete answer
  after:     button disabled until complete; 0 incomplete captures
             (and 1,620 taps, so users simply waited)

Users waited. Disabling the button cost 12% of copy taps and zero complaints.

One honest limit: the segment approach depends on the model emitting well-formed NDJSON, and it does not always — 2.1% of responses had a malformed segment in our measurement, which we handle by falling back to hold-until-complete for that response. A parser error becomes a slower answer rather than a wrong one, which is the right failure direction and is still a failure.

What this generalizes to#

Changing when a user sees something changes what you are claiming. Progressive rendering, optimistic updates, skeleton states, incremental search results, partial page loads. Each one creates intermediate states that did not previously exist, and each state is read as a statement.

A prefix of a valid output is not a valid output. Validating the final result tells you nothing about what was on screen on the way there. Where correctness matters, the intermediate states need their own review — ours failed 28% of the time.

Choose a render unit with semantic meaning. Tokens and characters are transport units. Sentences, segments, blocks and records are units someone can reason about. Streaming at the transport unit pushes the semantic question onto the renderer, which cannot answer it.

Gate capture on completion. Copy, share, save, submit, print — every action that freezes the current state must know whether the state is final. This is the easiest part to forget and the most visible when it goes wrong.

Related#

• Running a model on-device is a memory decision first — the other half of local inference
• StateFlow drops what you needed and keeps what you didn't — modelling partial and final state
• Your agent's tool output is untrusted input — another case of trusting a model's shape

Keep reading#

This is part of 100 Days of Coding Agents — building agents that are useful on real codebases, with the failure modes nobody puts in the demo. Follow MobileByteSensei for the rest.