How to know your on-device model got worse
A tiny eval suite that fails the build, the budget that bounds it, and why a verdict needs a third value meaning neither pass nor fail but undetermined.
evals
10 posts
Quantization has a quality cliff, and you can see it
Quantization degrades non-linearly, so an average score hides the cliff. Find it with a suite that fails per-case, and keep the per-case results, not the mean.
The agent that writes the check that blocks it
When the same agent writes the fix and the gate, the gate agrees by construction. RED before GREEN is what makes a machine-written check mean anything.
I swept the parameter instead of arguing about it
A threshold argument settled by measuring four values over the real corpus. The knee, the trade that stops paying, and why the numbers go in the comment.
A gate that fixes what it finds
An auto-healing loop must fix the source, never the symptom, and must never edit the thing that found the problem. Confidence floors and bounded rounds.
What to log when the actor is a model
Agent telemetry that survives the session: log the measurement rather than the verdict, and build a schema with no slot for the things that must not travel.
A verification step that can pass while the bug is present is not a verification step
Static checks read what was written; the defect lives in what was drawn. Four artifacts that passed every gate this session and failed the moment someone looked.
Asking AI to check its own work does not work. Ask it to disprove it.
A verifier asked to confirm will confirm. Inverting the instruction to 'try to prove this wrong' found bugs that three rounds of review had passed, including one in the verifier itself.
A test that cannot fail is not a test. Mine passed twice.
A RED test fixture that passed because a comment explaining the omission contained the omitted string. The failure mode is generic: any test whose input mentions what it removes.
100 technical posts, and the gates that stopped them being slop
Publishing at volume with an AI agent is easy. Publishing at volume without shipping slop needs machinery that refuses, and refuses loudly.