Updating a model without shipping an app release

A store review is days. A prompt change that fixes a bad output is minutes of work waiting behind it.

So the model stops being part of the binary and becomes something the app fetches — and that single decision moves you out of the release process and into running a content pipeline, with everything that implies.

The case this post is about#

One candidate model, through an overnight pipeline, stopping one step short:

{ staged: true, notes: [ "WARNING: 0.93 is below the live model's 0.94" ] }
{
  rolled: false,
  halt: "model-not-promoted",
  message: "2026.09 is staged and verified. Nobody has promoted it."
}

The candidate cleared the eval floor of 0.92. It is also worse than what is currently in production, by one point, and a pipeline gating on evalScore >= EVAL_FLOOR ships it — at 3am, to everyone, with that warning in a log.

Shipping weights as content buys a day instead of a week. This post follows that one candidate, because the thing it also buys is every problem a content pipeline has, and the warning above is why the last step is a refusal rather than a threshold.

The symptom#

The thing you want to change is rarely the weights. It is a prompt, a threshold, a post-processing rule, a list of refusal phrases. Small, surgical, and currently welded to a release train that assumes you are changing code.

Decoupling is obviously right and the cost is not obvious. The moment the model ships separately, three questions that the store used to answer for you become yours:

• which version is a given install actually running
• what happens when a fetch half-succeeds
• how do you get back to the previous one, quickly, without a release

The store answered all three badly but it did answer them. Now the binary is on one cadence and its behaviour is on another, and the app version no longer identifies the app's behaviour.

Why the obvious fix failed#

The obvious fix is to fetch the latest on launch and cache it. One endpoint, one version check, done.

It fails on partial state. A fetch that dies at 80% leaves a file that exists, has plausible size, and is not a model — and the failure surfaces later, somewhere unrelated, as a load error or garbage output. The download succeeded as far as anything local can tell.

The second attempt is to check a version number before using it. Better, and it verifies the label rather than the bytes. A truncated file with correct metadata passes.

The actual mechanism#

Four parts, and the fourth is the one people skip because it feels like paperwork.

Content-address the artifact. The thing you fetch is identified by the hash of its bytes, and you verify that hash after download, before use. This collapses did it download completely, is it the version I asked for and has it been tampered with into one check you cannot forget to do separately.

Stage, verify, then swap. Download to a staging path, verify, and only then make it the active one. Never write over the artifact that is currently working. The old one stays on disk until the new one has proven itself, which is also your rollback.

Record what each result was produced with. Every output carries the version that generated it. Not in a log — attached to the artifact, the way a provenance sidecar sits beside a generated asset. This is what makes a regression attributable instead of mysterious.

Put a human on the promote. Building a new model version and making it live for everyone are two actions, and the second is the one that cannot be taken back quietly.

The fix#

The part I would argue hardest for is the last one, because it is the one that gets automated away first and it is the one I have watched fail.

In this codebase the store listing and the app profile are deliberately separate, and promoting one to the other is an explicit human step. It exists because the default content is a template placeholder, and an automated promote would publish that placeholder to a live store listing. The gate's whole job is to refuse to proceed until a person names which variant ships.

The same shape applies to a model. The build pipeline can produce a candidate, verify it, run the eval suite and stage it. What it should not do is decide that everyone now gets it. A rollout is a judgement about blast radius and timing, and the fact that it could be automatic is exactly why it should not be.

Rolling forward to a bad model is survivable. Rolling forward to a bad model automatically, at 3am, to everyone, is the incident.

The pipeline that respects that boundary does every mechanical step and then stops with a named refusal, which is the only part of it a person has to read:

type Candidate = { version: string; sha256: string; sizeMb: number; evalScore: number };
type Promotion = { version: string; promotedBy: string; at: string } | null;

const EVAL_FLOOR = 0.92;

function stage(c: Candidate, current: Candidate): { staged: boolean; notes: string[] } {
  const notes: string[] = [];
  if (c.evalScore < EVAL_FLOOR) return { staged: false, notes: [`eval ${c.evalScore} below floor ${EVAL_FLOOR}`] };
  if (c.evalScore < current.evalScore) notes.push(`WARNING: ${c.evalScore} is below the live model's ${current.evalScore}`);
  if (c.sizeMb > current.sizeMb * 1.2) notes.push(`size +${Math.round((c.sizeMb / current.sizeMb - 1) * 100)}% over live`);
  return { staged: true, notes };
}

function rollout(c: Candidate, promotion: Promotion, percent: number) {
  // Everything above is mechanical and runs unattended. THIS is the human gate,
  // and its default is refusal — not because the pipeline is unsure, but because
  // blast radius and timing are not properties the pipeline can know.
  if (promotion === null || promotion.version !== c.version) {
    return {
      rolled: false,
      halt: "model-not-promoted",
      message:
        `${c.version} is staged and verified. Nobody has promoted it.
` +
        `  run:  model-promote --version ${c.version} --percent <n>
` +
        `  live: ${c.version === promotion?.version ? "?" : "previous"} model remains in service`,
    };
  }
  return { rolled: true, percent, by: promotion.promotedBy, at: promotion.at };
}

A candidate going through the whole pipeline overnight:

const live = { version: "2026.07", sha256: "71a4...", sizeMb: 28, evalScore: 0.94 };
const cand = { version: "2026.09", sha256: "0c57...", sizeMb: 31, evalScore: 0.93 };

console.log(stage(cand, live));
console.log(rollout(cand, null, 100));

{ staged: true, notes: [ "WARNING: 0.93 is below the live model's 0.94" ] }
{
  rolled: false,
  halt: "model-not-promoted",
  message: "2026.09 is staged and verified. Nobody has promoted it.
" +
           "  run:  model-promote --version 2026.09 --percent <n>
" +
           "  live: previous model remains in service"
}

The warning is why the gate has to be a refusal rather than a threshold. 0.93 clears the floor and is worse than what is running — a pipeline deciding on evalScore >= EVAL_FLOOR ships it, at 3am, to everyone, with the warning in a log. The halt puts the same two numbers in front of a person whose job it is to decide whether a point of eval score is worth whatever else the release contains.

How we proved it#

What I have actually run is the content version of this, which is this blog.

Posts are written, built, and deployed to a live site on their own cadence with no application release involved. Every generated asset carries a provenance sidecar recording what produced it, and the deploy is a separate explicit step from the build. The store-listing promote is a hard human gate that refuses to proceed on an unpromoted variant, for the reason above — the placeholder would otherwise go live.

build      produces a candidate, verifies it        automated
promote    decides everyone gets it                 human, gated

The census point is the one I can speak to most directly, because it caught a real defect. A field on every artifact recording which generator produced it is what revealed that a fallback path had produced 303 of 304 shipped items while the documentation described it as the exception. Nobody suspected it. The field made the question answerable by counting instead of believing.

Transfer that intact: if each output does not record the model version that produced it, then after a staged rollout you cannot answer which version produced this bad result — and that is precisely the moment you need to.

Not measured: I have not operated a model-delivery pipeline on a device fleet. Content-addressing, staged swap, provenance and the human promote are from shipping content this way; the inference specifics are reasoning.

What I expect to be harder with weights#

Three differences I would plan for rather than discover.

Size changes the failure mode. A post is kilobytes; a model is tens or hundreds of megabytes. Partial downloads stop being rare, resumption stops being optional, and "fetch on launch" becomes a background job with its own states the UI has to represent.

The staged copy is expensive. Keeping the previous model for rollback means holding two on a device where storage is the user's, not yours. The rollback property I argued for above has a real cost here that it does not have for text.

Compatibility is bidirectional. App code and model expect things of each other — tokenizer, output schema, prompt format. Decoupling the cadences means an old app can meet a new model, so the artifact needs to declare what app versions it is valid for, and the app needs to refuse one it cannot use rather than loading it and producing nonsense.

That last one is the one I would get wrong first, and it argues for the compatibility range living in the manifest beside the hash, checked at the same moment.

What this generalizes to#

When you move something out of the binary to escape the release cadence, you have not removed a release — you have created a second one with no process attached.

Give it the things the store was providing badly: an identity that is the bytes rather than a label, a staged swap that keeps the last working copy, a record on every output of what produced it, and a human who decides when everyone gets it. The first three are mechanical. The fourth is the one worth defending when somebody proposes automating it, because the question it answers is not is this build good but do we want this live right now — and only one of those is a property of the artifact.