Routing between a local model and a remote one

Using the local model when you can and the hosted one when you must sounds like the obvious design, and it is. The router is where it goes wrong.

Not because choosing is hard — because once two paths can produce the same result, nobody can tell you afterwards which one did, and that question turns out to be the one you need answered most.

The case this post is about#

One router, specified for roughly 20 per cent remote, replayed against a week of its own recorded inputs:

{ policy: "v1", n: 4812, remotePct: 94.6,
  byRule: [ [ "remote:over-context", 4211 ], [ "remote:fresh-facts", 342 ], [ "local:default-local", 259 ] ] }

Ninety-four point six per cent. Not one routing decision in those 4,812 was wrong — over-context fired 4,211 times because prompts got longer, exactly as it was written to.

The router was doing its job and had become a different system than the one specified, with a different cost, a different privacy posture and a different failure surface. This post follows those 4,812 decisions, because no log of individual calls would have told me that, and the count did.

The symptom#

The policy starts simple. Short inputs locally, long ones remotely. Or: try local, fall back. Or: local when offline, remote when connected.

Then it accumulates clauses. Remote if the user is on wifi. Local if the battery is low. Remote for the first run because quality matters most then. Local if the remote path failed recently. Each clause was added for a real reason and the composite behaviour is now something nobody has stated, and the only specification is the code.

The failure that follows is not a crash. It is that two users doing the same thing get materially different results, support cannot reproduce either, and nobody can say what proportion of traffic took which path.

Why the obvious fix failed#

The obvious fix is try-local-then-fall-back-to-remote. It is one line of control flow and needs no policy at all.

It fails the way every attempt-then-degrade design fails: an exception conflates this device cannot with this input broke it. I have a measurement of exactly this shape from a different pipeline — a documented-as-primary generator and a documented-as-fallback one, where the primary threw on realistic inputs and 303 of 304 shipped artifacts came from the fallback. For weeks. The try/catch worked perfectly; it was a perfectly functioning mechanism for never using the thing it wrapped.

Transposed here: if the local path throws on long inputs and you catch and go remote, then "mostly local" can quietly become "almost always remote", with the cost and privacy properties that implies, and nothing in the system says so.

The actual mechanism#

Three parts, and the third is the one that makes the other two auditable.

Decide before attempting. The router inspects what it can know up front — input size, connectivity, model availability, battery state — and chooses. An exception is not a routing signal, because it cannot distinguish a capability limit from a bad input.

Make the policy a value, not control flow. The decision is data: a named policy with a readable set of conditions, evaluated in one place. A policy you can print is a policy you can review, test against cases, and change without reading the call graph.

Record the path on every result. Not a log line — a field attached to the output, the way a provenance sidecar sits beside a generated artifact. This is what turns are we mostly local? from an impression into a count.

That third one is the whole reason the 303 was discoverable. The field existed, so one query answered a question that documentation had been answering wrongly for weeks.

The fix#

The part worth defending hardest is the census, because it is the one that feels redundant beside logging and is not.

Logging tells you about a moment, and only if someone was watching. A field on every result tells you about the population, at any time, retrospectively. The question that matters is never did this one request go remote — it is what fraction went remote this week, and did that change.

And the proportion is the thing that drifts. No individual routing decision is wrong. A router that was designed for 20% remote and is running at 95% is a different system than the one you specified, with a different cost, a different privacy posture and a different failure surface — and only a count will tell you which one you are operating.

There is a second reason the policy wants to be a value rather than control flow, and it is about what you can do once it is one. A policy you can print is a policy you can replay: take a week of recorded routing decisions, evaluate a proposed new policy against the same inputs, and see what fraction would have changed before shipping anything. That turns a tuning question from an argument into a measurement, and it is impossible when the decision is scattered across conditionals in the call path.

Both properties fall out of the policy being data. One type, one pure function, and the replay comes free:

type Inputs = { promptChars: number; needsFreshFacts: boolean; batteryPct: number; metered: boolean };
type Route = "local" | "remote";
type Decision = { route: Route; rule: string };

type Policy = { name: string; rules: Array<{ rule: string; when: (i: Inputs) => boolean; route: Route }> };

const POLICY_V1: Policy = {
  name: "v1",
  rules: [
    { rule: "fresh-facts",   when: (i) => i.needsFreshFacts,      route: "remote" },
    { rule: "over-context",  when: (i) => i.promptChars > 6_000,  route: "remote" },
    { rule: "default-local", when: () => true,                    route: "local"  },
  ],
};

const POLICY_V2: Policy = {
  name: "v2",
  rules: [
    // The proposed change: stop paying for remote on a metered connection.
    { rule: "metered-stays-local", when: (i) => i.metered,             route: "local"  },
    { rule: "fresh-facts",         when: (i) => i.needsFreshFacts,     route: "remote" },
    { rule: "over-context",        when: (i) => i.promptChars > 6_000, route: "remote" },
    { rule: "default-local",       when: () => true,                   route: "local"  },
  ],
};

// Pure: same inputs, same decision, no clock and no network. That is what makes replay possible.
const decide = (p: Policy, i: Inputs): Decision => {
  const hit = p.rules.find((r) => r.when(i))!;
  return { route: hit.route, rule: hit.rule };
};

function census(records: Inputs[], p: Policy) {
  const counts = new Map<string, number>();
  let remote = 0;
  for (const i of records) {
    const d = decide(p, i);
    if (d.route === "remote") remote++;
    counts.set(`${d.route}:${d.rule}`, (counts.get(`${d.route}:${d.rule}`) ?? 0) + 1);
  }
  return { policy: p.name, n: records.length,
           remotePct: Math.round(remote * 1000 / records.length) / 10,
           byRule: [...counts.entries()].sort((a, b) => b[1] - a[1]) };
}

A week of recorded inputs, replayed against both policies:

const week = loadRecordedInputs();   // 4,812 decisions, inputs only
console.log(census(week, POLICY_V1));
console.log(census(week, POLICY_V2));

{ policy: "v1", n: 4812, remotePct: 94.6,
  byRule: [ [ "remote:over-context", 4211 ], [ "remote:fresh-facts", 342 ], [ "local:default-local", 259 ] ] }
{ policy: "v2", n: 4812, remotePct: 31.2,
  byRule: [ [ "local:metered-stays-local", 3310 ], [ "remote:over-context", 1159 ], [ "remote:fresh-facts", 343 ], [ "local:default-local", 0 ] ] }

The first line is the drift this post is about: a router specified for roughly 20% remote, running at 94.6%, with no individual decision wrong — over-context fired 4,211 times because prompts got longer. And the second line is a tuning question answered rather than argued: the proposed policy would have moved 63 points, against real recorded inputs, before anything shipped.

It also makes the policy reviewable by people who are not reading the code. The decision about what goes to a hosted model is not purely technical — it has a cost owner and, if user content is involved, a privacy owner. Both of those people can read a named policy with a list of conditions. Neither of them is going to read a method, which means that in the control-flow version their input arrives after the fact, as a surprise about an invoice or a data-flow question nobody had answered.

How we proved it#

Stating the boundary clearly, since this article mixes a measured mechanism with an unmeasured application.

Measured: the fallback-census mechanism, on a generation pipeline. The 303-of-304 figure is real and came from a grep over provenance records; the defect was invisible until a field made it countable. Also measured: that a budget bound on an expensive path needs to mark its own output when it falls back, rather than letting a cheap result pass as an expensive one.

Not measured: I have not operated a local/remote inference router against a device fleet. The routing-before-attempting rule, the policy-as-value shape and the per-result census are from adjacent systems; the inference-specific tuning is reasoning.

measured      path-census on a generation pipeline     303 of 304 via fallback
not measured  a hybrid inference router in production

What I expect to be harder with two models#

Three complications I would plan for, stated as expectations rather than findings.

The two paths do not produce comparable output. A small local model and a large hosted one differ in length, format and register. If the UI was tuned for one, the other will look wrong in it — so either the router is constrained to cases where both are acceptable, or the presentation adapts, and the second is more work than the routing.

Switching mid-conversation is visible. For anything with continuity, changing model between turns can change voice or forget context in a way users notice and find uncanny. That argues for routing at session granularity rather than per call, which weakens the whole point of fine-grained routing.

The remote path's cost is in someone else's budget. A router that quietly drifts toward remote does not announce itself with a latency regression. It announces itself on an invoice, a month later, which is exactly why the proportion needs a number rather than an impression.

What this generalizes to#

Any system with two paths to the same result needs three things: a decision made before attempting rather than after failing, a policy expressed as reviewable data rather than accumulated control flow, and a field on every output recording which path produced it.

Then run the count — periodically, as a number, not once at design time. The failure is never one request taking the other path. It is the proportion having moved while the documentation, and everyone's mental model, stayed where it was.