Your prompt is a config file, and it needs the same discipline
Untracked prompt edits are untracked behaviour changes.
TL;DR
A prompt is an input that determines production behaviour. Edited in a dashboard, it has none of the controls you apply to the code it replaces — no version, no diff, no review, no eval gate, no rollback.
Ours regressed a classification agent from 94% to 71% accuracy because someone removed a sentence that looked redundant. It took nine days to find, because there was no record that anything had changed.
Treat the prompt as what it is: a config file that changes behaviour. In git, reviewed, gated by evals, versioned in the artifact, and attributable in every output.
The case this post is about
A support-ticket classifier, with its prompt in the vendor dashboard where it was easy to iterate:
You are a support ticket classifier. Classify each ticket into exactly one of:
BILLING, TECHNICAL, ACCOUNT, FEEDBACK, OTHER.
Consider the ticket's primary intent, not incidental mentions. A ticket that
mentions a payment while asking how to export data is TECHNICAL, not BILLING.
Respond with only the label.
That second paragraph is the interesting one. It was added in month two after a batch of misclassifications, and it raised accuracy from 86% to 94%.
The symptom
day 0 classification accuracy 94.1%
day 1 accuracy 71.3%
day 2–8 accuracy 70–73%, rising ticket-routing complaints
day 9 cause found
Twenty-three points, overnight, with no deploy. The investigation went through every other possible cause first:
day 1-2 model version change? vendor confirmed no change
day 2-4 input distribution shift? ticket mix compared against 30 days: no change
day 4-6 retrieval/context change? no code deploys in the window
day 6-8 a provider-side regression? escalated to the vendor, nothing found
day 9 someone remembered editing the prompt "to tidy it up"
The edit:
You are a support ticket classifier. Classify each ticket into exactly one of:
BILLING, TECHNICAL, ACCOUNT, FEEDBACK, OTHER.
-Consider the ticket's primary intent, not incidental mentions. A ticket that
-mentions a payment while asking how to export data is TECHNICAL, not BILLING.
-
Respond with only the label.
The paragraph had looked like verbose instruction-following boilerplate. It was carrying the single most valuable piece of guidance in the prompt, and the person who removed it had no way to know that — because the reason it existed was not recorded anywhere.
Three properties made nine days possible:
No diff. The dashboard stored the current text and an edit timestamp. No previous version, no author, no reason.
No version in the output. Classifications carried a model name and no prompt identifier, so it was impossible to correlate a result with the prompt that produced it.
And no eval gate. The change went live immediately. A 60-example eval would have caught it in two minutes.
Why the obvious fix failed
"Restrict dashboard access." We did. It means prompt changes now require asking one of three people, so iteration slows and the changes still have no diff or gate — the bottleneck replaced the control.
"Add a changelog field to the dashboard." A free-text box that is honestly filled in sometimes. It helps the archaeology and gates nothing.
"Copy the prompt into the code." The right direction, and our first version inlined it as a string constant, which lost the ability to iterate without a deploy. That is a real capability — prompt iteration is fast and should stay fast.
"Have the vendor dashboard keep history." Some do, and it is history of the text without the eval result, the author's reasoning, or a link to the code that consumed it.
The actual mechanism
A prompt is a production input that determines behaviour, and the controls you have for such inputs depend entirely on where they live:
code → git, review, CI, eval, canary, rollback, bisect
config in git → git, review, CI, eval, rollback
config in a UI → an edit timestamp
prompt in a UI → an edit timestamp
The prompt replaces code — it is the classification logic — and it sits in the column with the fewest controls.
Moving it into the first row is mostly a loading change: read the prompt from a versioned file, hash it, and stamp that hash on every output so a classification can be traced back to the exact text that produced it.
import hashlib, pathlib
class PromptRegistry:
def __init__(self, root: pathlib.Path):
self.root = root
self._cache: dict[str, tuple[str, str]] = {}
def load(self, name: str) -> tuple[str, str]:
"""Returns (text, version). Version is content-derived, so it CANNOT drift."""
if name in self._cache:
return self._cache[name]
path = self.root / f"{name}.md"
if not path.exists():
# Never fall back to a default or a previous version. A missing prompt is a
# deploy error; silently classifying with the wrong text is the whole post.
raise FileNotFoundError(f"no prompt '{name}' in {self.root} — refusing to classify")
text = path.read_text()
version = hashlib.sha256(text.encode()).hexdigest()[:12]
self._cache[name] = (text, version)
return text, version
def classify(ticket, registry, model):
prompt, version = registry.load("ticket_triage")
result = model.complete(prompt.format(ticket=ticket.body))
return {
"ticket": ticket.id,
"label": result.label,
"prompt_version": version, # ← the column that made the 8 days searchable
"model": model.name,
}
Replayed over the eight days of stored classifications, once the field existed:
prompt_version first seen n p(bug) p(billing)
a3f1c9e4bb20 2026-08-02 09:14 11,402 0.31 0.22
7d02e8cc1a5f 2026-08-06 16:48 3,160 0.29 0.41 ← the regression
a3f1c9e4bb20 2026-08-09 10:02 2,884 0.31 0.22 ← reverted, matches the original
The hash is doing the work a deploy id does for code. Two of those rows being byte-identical is
what turned "something changed around the 6th" into "this exact text changed, here is the diff,
and reverting it restored the distribution" — a git diff between two hashes, rather than a week
of asking people what they remembered editing.
Three specific gaps, each mapping to a day of the investigation:
No diff means no bisect. The standard move for a regression with no deploy is to bisect deploys. With the behaviour-determining input outside version control, the search space does not include the actual change.
No version in the output means no correlation. We had 8 days of classifications and could not partition them by prompt, so we could not see the change point in the data even though it was there.
And no gate means no feedback. The author got no signal that the change was harmful. From their side it was a tidy-up that did nothing.
A prompt is the logic. Keeping it outside version control is keeping your logic outside version control, and every control you rely on for code is unavailable by construction.
The fix
Prompts in git, loaded at runtime, gated by evals, and stamped into every output.
1. The prompt is a file, with its reasoning alongside it:
# prompts/ticket_classifier/v7.yaml
id: ticket_classifier
version: 7
eval_suite: evals/ticket_classification.yaml
min_accuracy: 0.90 # the gate
system: |
You are a support ticket classifier. Classify each ticket into exactly one of:
BILLING, TECHNICAL, ACCOUNT, FEEDBACK, OTHER.
# DO NOT REMOVE: added v3, raised accuracy 86% → 94% (eval run 2026-08-14).
# Tickets commonly mention a payment incidentally; without this they are
# misclassified as BILLING.
Consider the ticket's primary intent, not incidental mentions. A ticket that
mentions a payment while asking how to export data is TECHNICAL, not BILLING.
Respond with only the label.
The comment is the fix for the human half: the next person to see that paragraph as redundant now sees why it exists and what removing it costs.
2. Loaded at runtime from a ref, so iteration stays fast without a deploy:
# the prompt is fetched from a git ref, cached, and hot-reloadable
prompt = prompt_store.load("ticket_classifier", ref=os.environ["PROMPT_REF"])
Changing PROMPT_REF to a branch is a config change; the prompt content itself is reviewed. You
keep the iteration speed and lose the untracked edit.
3. An eval gate in CI on the prompt directory:
# .github/workflows/prompts.yml
on: { pull_request: { paths: ["prompts/**"] } }
jobs:
eval:
steps:
- run: python -m evals run --prompt prompts/ticket_classifier/v7.yaml --fail-under 0.90
$ python -m evals run --prompt prompts/ticket_classifier/v7.yaml
examples: 240
accuracy: 0.712
❌ accuracy 0.712 below min_accuracy 0.90
exit 1
Two minutes, on the pull request. That is the control the nine days bought.
4. And the version stamped into every output:
result = Classification(
label=label,
prompt_id="ticket_classifier",
prompt_version=7,
prompt_sha=prompt.sha, # content hash, so a hand-edit is detectable
model=model_id,
)
-- the query that would have answered day 1 in thirty seconds
SELECT prompt_sha, COUNT(*), AVG(CASE WHEN label = human_label THEN 1 ELSE 0 END) AS accuracy
FROM classifications WHERE created_at > now() - interval '14 days'
GROUP BY prompt_sha ORDER BY MIN(created_at);
prompt_sha n accuracy
a3f1c9… 38,210 0.941
7b22e4… 9,880 0.713 ← change point, visible immediately
5. Plus a canary, since the eval suite cannot cover everything:
rollout:
stages: [{ percent: 5, hold: 1h }, { percent: 50, hold: 4h }, { percent: 100 }]
abort_on: { metric: classification_accuracy, threshold: 0.90 }
How we proved it
That the eval catches the exact regression, replayed against the removed paragraph:
prompt v6 (with the paragraph): accuracy 0.941 ✓
prompt v7-candidate (removed): accuracy 0.712 ✗ BLOCKED
That the version stamp makes the change point visible:
time to identify the cause, on a simulated prompt regression
before (no stamp, no diff): 9 days (actual)
after: 4 min (drill, twice)
And the eval suite's own coverage, because a gate is only as good as its examples:
evals/ticket_classification.yaml
examples: 240
drawn from: hand-labelled production tickets
class balance: BILLING 52 · TECHNICAL 61 · ACCOUNT 44 · FEEDBACK 39 · OTHER 44
includes the incidental-payment case explicitly: 18 examples
Those 18 examples exist because of this incident. The general lesson there: every regression you suffer should become an eval case, which is the same discipline as a regression test.
The adoption cost, measured over the following quarter:
prompts migrated to git: 11
median time to migrate one: 25 min
prompt changes in the quarter: 34
blocked by the eval gate: 6 ← 6 regressions caught pre-merge
changes still shipped same-day: 31/34 (iteration speed retained)
Six caught regressions in a quarter, with same-day shipping preserved on 31 of 34 changes. The three that took longer were the ones that needed new eval examples, which is the right reason to be slow.
One limit worth naming: the gate is --fail-under 0.90 on an aggregate, which cannot catch a change
that trades accuracy between classes while holding the total — we had one of those (BILLING recall
down 9 points, OTHER up) that passed the gate and was caught by the canary. Per-class thresholds
would help and we have not added them; the canary remains the backstop.
What this generalizes to
Anything that determines production behaviour belongs in version control. Prompts, config, feature flags, routing rules, model choices, ML feature definitions, SQL in a dashboard. The test is not "is it code" but "does changing it change what users get".
Record why a line exists, next to the line. The removed paragraph looked redundant because its justification lived in a chat thread from four months earlier. A two-line comment with the eval numbers is what makes a non-obvious instruction survive a tidy-up.
Stamp the version of every behaviour-determining input into your output. Model, prompt, config revision, feature-flag snapshot. Without it you cannot partition results by what produced them, which is the difference between a thirty-second query and a nine-day investigation.
Turn every regression into an eval case. The 18 incidental-payment examples exist because of this incident, and they make the same mistake impossible to ship twice. This is regression testing, applied to a component whose logic happens to be prose.
Related
- Context is a budget, and what you load first wins — measuring what the prompt loads
- The eval that passed because the function returned a plausible stub — making the eval trustworthy
- The config change that was really a deploy, and had none of the safeguards — the same gap, for config
Keep reading
This is part of Context engineering for code — what to put in a model's context, measured rather than assumed. Follow MobileByteSensei for the rest.
One post a week, no digest
New writing when it lands. No roundups, no course pitch, no "10 tips".