Core ML and LiteRT solve the same problem differently

A platform-native inference runtime and a portable one are solving the same problem with different priorities, and the difference shows up in your code as a seam you have to design.

The interesting question is not which is better. It is what a shared declaration over both of them is allowed to promise — because the compiler will enforce the signature and nothing will enforce the behaviour.

The case this post is about#

One text generator, running on two inference runtimes, and the question of where the line between shared and platform code goes:

                      could it live above?   would divergence fail loudly?   verdict
run(tokens)           no — platform call     yes, crash/shape mismatch       BELOW
close()               no — platform handle   yes, leak is observable         BELOW
encode/decode         yes — table lookup     NO, plausible wrong tokens      above
sample()              yes — arithmetic       NO, plausible wrong completion  above
tokenCount()          yes — arithmetic       NO, plausible wrong truncation  above

Six candidates for the shared declaration, two of which belong there. The three marked NO on the second question are the ones a single test would have let through, and they are the dangerous ones: two tokenizers disagreeing by one token produce two plausible completions and no failure anywhere.

This post follows those six rows, because the first question alone — is this platform-ish? — puts all three below the line.

The symptom#

You want one inference API in shared code and two implementations underneath. The structure is obvious, the compiler checks it, and the thing it does not check is semantic parity.

Two implementations can satisfy the same signature while differing in ways that decide whether shared code is correct:

• one is fast on first call, the other pays a compilation or warm-up cost
• one throws on an unavailable resource, the other returns empty
• one supports cancellation, the other ignores it
• one is safe to call concurrently, the other serialises internally
• one owns its model lifecycle, the other expects you to

Every one of those type-checks. Shared code written against the declaration is correct on the platform its author tested and subtly wrong on the other, with no signal until a device runs it.

Why the obvious fix failed#

The obvious fix is a richer shared interface: expose the differences, add capability flags, surface the knobs.

That migrates the platform difference into the shared declaration, which is the worst place for it. Every caller now reasons about both runtimes, and the seam has stopped separating anything — it is a union of two APIs with a keyword in front.

The second attempt is the opposite: keep the declaration minimal and let each side do what it can. That pushes the difference into behaviour, which is worse, because behaviour is not inspectable at the call site.

The actual mechanism#

Put everything that is not the platform call itself above the line, and let the declaration promise only what both sides can genuinely honour.

The platform side should own almost nothing: load, run, release. Building the input, validating shape, timing, retry policy, mapping to domain types, the state machine around availability — all of that is written once, tested once, identical on both platforms.

Two properties make the declaration honest:

Return a value rather than throwing. Failure is then something both sides can produce, so the difference between throws and returns empty never reaches a caller.

Promise nothing about timing. A suspending call says "this may take a while" on both sides, which is the only timing claim that survives two very different runtimes.

And one property specific to inference: the model's lifecycle is explicit in the declaration, because the two runtimes will not agree about when a loaded model is released, and shared code that assumes one behaviour will leak or reload on the other.

The fix#

The rule that does the work: the shared declaration should be the narrowest thing that still lets common code do its job.

Every method is a promise two independently-written implementations must keep in step, with nothing checking that they did. Two methods, two chances to diverge. Ten, ten — and the divergences are invisible.

The test when adding anything: could I write this above the line instead? Usually yes, because most of what surrounds a platform call is arithmetic and state. What remains below is the irreducible call.

There is a second test, and it catches the additions the first one waves through: if the two sides answered this differently, would anything fail loudly? A method whose divergence produces a crash is survivable — somebody gets a stack trace. A method whose divergence produces a plausible wrong result is the dangerous kind, because it reaches a user as a quality problem rather than a bug, and nothing in the build will ever point at the seam.

Applied, the seam comes out at two methods, and everything that looked like it belonged below the line turns out to be arithmetic:

// BELOW the line: the irreducible platform call, and nothing else.
expect class InferenceEngine(modelPath: String) {
    suspend fun run(inputTokens: IntArray): FloatArray      // raw logits
    fun close()
}

// ABOVE the line: shared, written once, identical on both platforms. Every one of
// these was a candidate for the expect declaration and none of them needs to be.
class Inference(private val engine: InferenceEngine, private val vocab: Vocab) {

    suspend fun complete(prompt: String, maxTokens: Int, temperature: Float): String {
        val out = StringBuilder()
        var tokens = vocab.encode(prompt)                   // arithmetic

        repeat(maxTokens) {
            val logits = engine.run(tokens)                 // the ONE platform call
            val next = sample(logits, temperature)          // arithmetic
            if (next == vocab.eos) return out.toString()
            out.append(vocab.decode(next))
            tokens = tokens + next                          // state
        }
        return out.toString()
    }

    // Divergence here would be SILENT on both tests: two platforms sampling
    // differently produce two plausible completions and no failure anywhere.
    private fun sample(logits: FloatArray, temperature: Float): Int {
        if (temperature <= 0f) return logits.indices.maxBy { logits[it] }
        val scaled = logits.map { it / temperature }
        val max = scaled.max()
        val exp = scaled.map { kotlin.math.exp(it - max) }
        val sum = exp.sum()
        var r = nextFloat() * sum
        exp.forEachIndexed { i, e -> r -= e; if (r <= 0f) return i }
        return exp.lastIndex
    }
}

The candidate declarations, scored against both tests:

                      could it live above?   would divergence fail loudly?   verdict
run(tokens)           no — platform call     yes, crash/shape mismatch       BELOW
close()               no — platform handle   yes, leak is observable         BELOW
encode/decode         yes — table lookup     NO, plausible wrong tokens      above
sample()              yes — arithmetic       NO, plausible wrong completion  above
tokenCount()          yes — arithmetic       NO, plausible wrong truncation  above
isAvailable()         no — platform query    yes, feature simply absent      BELOW?  -> above, derived from run()

Six candidates, two below the line. The three marked NO on the second test are the ones the first test alone would have let through as "platform-ish" — and they are precisely the dangerous ones: two tokenizers disagreeing by one token, or two samplers with different floating-point ordering, produce output a reviewer reads as a quality difference between platforms rather than as a bug with a location.

That second test is why run returning a value rather than throwing matters more than it looks. Exceptions are the mechanism by which a platform difference becomes invisible: one side throws, a caller somewhere catches broadly, and the behaviour on that platform quietly becomes the degraded path without any code saying so. I have watched exactly that happen in a different context — an attempt-then-catch design where the primary path threw on realistic input, so 303 of 304 shipped artifacts came from the fallback while every document described it as the exception. Values do not prevent divergence. They prevent divergence from being silent.

How we proved it#

Measured: the seam discipline itself, across a codebase where platform bridges exist for permission requests, biometrics, deep links, share sheets, calendar events and notification scheduling, implemented across six source sets. The narrow-declaration rule is what keeps those maintainable, and the failures have been exactly the ones above — a method added because one platform made it easy, and a divergence nobody caught until a device ran it.

Also measured, and load-bearing: where these modules live. A fork's own module declarations belong in a file the upstream sync does not overwrite; thirty-six of them were sitting in one it replaces wholesale, one sync away from silent deletion with no fork-owned file showing a diff. That is not an inference problem, it is a platform-module problem, and anything behind an expect/actual seam inherits it.

Not measured: I have not shipped against Core ML and LiteRT. I am deliberately not comparing their APIs, formats, operator coverage or performance, because I have not used them and a confident comparison of two runtimes I have not run would be exactly the plausible-sounding wrong thing I try not to publish. What I can tell you is what the seam between any two such runtimes has to survive.

Where I expect the seam to leak#

Model lifecycle. A session is not just a call — the model is loaded, possibly evicted under memory pressure, reloaded. If eviction differs across platforms, a failure value is not enough; callers need to distinguish this input failed from the model is gone, and that distinction will want to be in the declaration.

Streaming. One suspending call returning one result is the wrong shape for token streaming, which is the common case for text generation. The honest version is probably a flow, and I do not know whether both runtimes can honour the same cancellation semantics on it.

First-call cost. A suspending function does not distinguish a 20ms call from a multi-second first load. Shared code that cannot tell those apart makes bad decisions about what to show, and the fix is probably an explicit warm-up in the declaration — exactly the kind of addition the narrowness rule argues against.

That last one is a genuine tension rather than a detail, and I would rather flag it than present a seam that has not met the problem.

What this generalizes to#

When one API has two platform implementations, decide what the shared declaration may promise before writing either side. Keep it to the irreducible call; put the arithmetic, the state machine and the policy above the line where they are written and tested once.

Prefer values over exceptions and make no claims about timing, because those are the two places two runtimes diverge without the compiler noticing. Then treat every addition to the shared declaration as a promise you have just made twice, with nothing checking that both were kept.