Pin a commit, not a branch, when someone rehosts your images

Images for these posts live in a separate public repository and are served over raw file URLs. The site renders them. So does Medium — except Medium does not render them. Medium fetches them once, copies them onto its own CDN, and renders the copy, permanently.

That one difference decides which kind of URL you are allowed to write down.

The case this post is about#

One asset URL, handed to a platform that copies bytes exactly once:

sync files into the mirror
commit
push
read the new HEAD sha          ← the pin, derived
rewrite the single pin value
fetch one real asset URL       ← the proof, with backoff

Before the pin existed, that URL referenced a branch. The platform fetched it at create time, rehosted whatever was there that second, and kept that copy permanently — so an asset regenerated an hour later was correct on the site and wrong in the published copy, with nothing reporting a disagreement.

This post follows that one URL, because a branch ref is a promise to keep changing and the consumer had made a promise never to look again.

The symptom#

The first version is the one everybody has seen: you reorganise the asset repo, a file moves, and every URL pointing at the old path 404s. Unremarkable, and it is not the interesting failure.

The interesting one is that the breakage lands on some surfaces and not others, and the split is decided by a thing you cannot see — when each consumer happened to fetch.

A branch-pinned URL means whatever this branch holds right now. Your own site asks for it on every page load, so it always shows the current bytes. Medium asked for it once, at the moment the draft was created, and it is still showing that answer. Same URL, two surfaces, two different images, and neither one is wrong about the URL it was given.

Which means a mistake has a half-life rather than a moment. Replace an image and the site updates; the twenty posts already on Medium keep the old one. Fix a typo in a diagram and the fix lands everywhere the bytes are read live and nowhere they were copied. There is no error, no log line, and nothing in either system that knows the other exists.

The second version is worse because it is unrecoverable. If the URL is not serving yet at the instant the rehosting consumer fetches it, the copy it keeps is the failure. For Medium specifically: the API creates posts and does not update them, and does not delete them. There is no second attempt. A draft created against a URL that 404'd for four seconds has an image that will never load, and the only remedy is to abandon the post and create another one — which means a new URL, and the old one still live and broken.

So the blast radius is asymmetric in a way that matters for how hard you guard it. A wrong pin on your own site is a cache purge. The same wrong pin handed to a one-shot rehost is permanent.

The third version is the one that wasted my afternoon. You test the URL, it works, you publish, and it is broken for everyone else. Your browser had it cached from the push. The CDN edge nearest you had it; the edge the platform's fetcher uses did not. A single successful fetch from your own machine is the weakest possible evidence about a distributed cache, and it is the evidence everybody reaches for first because it is the only one that costs nothing.

And there is a fourth that only shows up with more than one person, or with one person and two machines. If the pin is written by hand, two people working the same day write two different pins, and both are correct at the moment of writing. The posts then carry a mixture, and reconstructing which image each post actually points at means reading every post rather than reading one config value.

Why the obvious fix failed#

Don't move files. The first and most reasonable answer, and it fails for the usual reason: it is a rule enforced by memory, the failure is silent, and the thing it protects is other people's copies of your bytes. It also does not help at all with the propagation case, where no file moved and the URL still 404'd.

Add a cache-busting query parameter. This is the attempt that feels most like a fix and is the most instructive failure, because it solves a different problem. ?v=2 gets you freshness: the next fetch bypasses whatever was cached. It does nothing for immutability — the URL still resolves through a branch, so when the branch moves again, ?v=2 serves the new bytes to anyone who reads it live. You have made the URL harder to cache without making it mean anything more specific.

Freshness and immutability look like the same property from a distance and they are opposites. Freshness asks am I seeing the latest? Immutability asks will this always be what it is now? A branch ref gives you the first and refuses the second. Reaching for a cache-buster when you need immutability is treating a stale read as the problem, when the problem is that the URL was never specific in the first place.

Make the consumer retry. Fine when the consumer is yours. Medium's fetcher is server-side, runs once, at a time you do not choose, with a timeout you cannot configure. There is no hook. Any reliability you want has to exist before the URL is handed over, because afterwards you have no participation in the transaction at all.

Sleep after pushing. The honest version of the propagation fix, and it is a guess dressed as a remedy. You are waiting on a cache you do not own, with no stated SLA, and the number you pick is either too short on the bad day or wasted on every good one. A sleep also cannot fail — it is a delay that reports success whether or not the thing you were waiting for happened.

The actual mechanism#

Use a content-addressed ref, and make the pin something a program writes rather than a person.

A commit SHA cannot serve different bytes than it served yesterday. That is not a convention or a cache policy — it is what the hash is. A branch name is a pointer that is expected to move; a commit is the thing it points at. So the choice of ref is a statement about whether the URL is allowed to change meaning, and for anything a consumer will copy and keep, the answer is no.

Three parts make it work in practice.

The pin is derived, never typed. After pushing the asset repo, read the new HEAD and rewrite every reference from that one value. A pin that is typed is a pin that is sometimes stale, and the staleness is invisible because a stale SHA is a perfectly valid URL serving perfectly real bytes — just last week's.

The pin lives in exactly one place. One value, consumed everywhere. The alternative — SHAs inlined across posts — makes the question "what does this site point at" unanswerable without a full scan, and guarantees a mixture the first time two changes overlap.

The pin is proven before it is used. This is the part that is not in anyone's documentation. A successful git push means the remote accepted the objects and moved the ref. It says nothing about whether the raw-file CDN in front of that repo will serve the new SHA on the next request, because that is a different system with its own propagation. Every other step in the chain reports failure with a non-zero exit. Propagation reports nothing at all — it is simply not ready yet, and the way you find out is a 404 that looks exactly like a wrong path.

So the proof is a real fetch: take one actual asset URL built from the new pin, request it, and retry with backoff until it returns. Not the repository root, not the API — a 200 on either tells you the repo exists, which was never in question. The thing you need to know is whether an object is served, so the check has to ask for an object.

The fix#

The ordering is the whole implementation, and every step earns its place:

sync files into the mirror
commit
push
read the new HEAD sha          ← the pin, derived
rewrite the single pin value
fetch one real asset URL       ← the proof, with backoff
publish

What makes this cheap is that there is no state to maintain. No version counter, no manifest of hashes, nothing that can drift out of sync with reality — the SHA already exists, git already knows it, and the only work is reading it instead of remembering it.

As a script, with the fetch where it belongs — after the push, before the publish:

sync_assets_into "$MIRROR"
( cd "$MIRROR" && commit_all "chore(assets): sync" && push_current_branch )

# The pin is DERIVED. No counter to bump, nothing to keep in step.
sha=$( cd "$MIRROR" && resolve_head )
probe="https://raw.example.com/$REPO/$sha/posts/$SLUG/header.jpg"

# The one transition with no local signal: ref moved, object SERVABLE. CDN
# propagation is seconds-to-minutes, so this needs backoff, not a single try.
for attempt in 1 2 3 4 5; do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 25 "$probe")
  [ "$code" = "200" ] && break
  sleep $(( attempt * 10 ))
done
[ "$code" = "200" ] || { echo "pin $sha not servable (HTTP $code) — refusing" >&2; exit 4; }

write_pin "$sha"
publish_post

The backoff is not politeness — it is the measured shape of the failure:

$ ./pin-and-publish.sh
  pushed  243a9b3d…
  probe   404   (attempt 1)
  probe   404   (attempt 2)
  probe   200   (attempt 3, +30s)
  pinned  243a9b3d
  published

A single-attempt check would have failed a perfectly good push. No check at all would have published a post whose images 404 — and the remote copy rehosts them at publish time, permanently.

The step people cut is the fetch, because it is the only one that looks redundant. Everything before it succeeded; what is left to check? That instinct is exactly inverted. The steps that already reported success are the ones you do not need to re-verify. The fetch exists precisely because the transition it covers — ref moved, object servable — is the one transition in the chain with no success signal of its own.

There is also a decision about what happens when the proof fails after its retries are spent. The tempting answer is a warning, and it is wrong here for the same reason it is wrong everywhere: the next step hands a URL to a system that cannot be corrected. A warning in front of an irreversible action is a decoration. It has to refuse.

How we proved it#

The rehost is real, and that is the fact everything else rests on. A published post's rendered markup points at the platform's own CDN host, not at the raw URL that was supplied. The platform fetched once and kept a copy. That is directly observable on any published post and it is the difference between a URL being a reference and a URL being a one-time delivery.

The pin is single-valued and derived. One hundred posts are live with images resolved through one commit-pinned base, written from the push's HEAD rather than by hand. The thing that makes that checkable is that there is exactly one value to look at.

And here is the part I want to be straight about: in the runs I have done, the propagation proof has passed on its first attempt every time. I have not observed the 404 it guards against. I cannot give you a propagation latency, because I have not measured one — what I have is a mechanism with no success signal sitting immediately before an action that cannot be undone.

That is a guard justified by blast radius rather than by observed frequency, and I think that is a legitimate reason to ship one, as long as it is stated plainly rather than implied to be battle-tested. The alternative is a check I only add after the first permanently broken post, and on a create-only API the first one is not recoverable. A guard that never fires costs a few hundred milliseconds per publish. The failure it covers costs a post.

What I have not solved: the old pins. Posts published against an earlier SHA still reference it, which is correct — their images are frozen and should be — but it means the asset repo can never garbage-collect a file any live post points at, and nothing currently tracks which SHAs are still referenced. Today that is fine because nothing has been deleted. It is a cost that accrues silently and I have not built the thing that would measure it.

A second limitation: this all assumes the asset host keeps serving historical commits. That is how the host behaves and I have relied on it, but it is a property of someone else's product, not a guarantee I hold. If it ever stopped, every published post would break at once and the immutable pin would be the reason — a mutable ref would at least have followed the repo somewhere else. I would rather name that than pretend the content-addressed version has no exposure.

What this generalizes to#

For every consumer of an asset you publish, ask one question: does it read every time, or copy once?

Readers want a mutable ref — they should track your latest. Copiers must be handed an immutable one, because the URL you give a copier is not a reference, it is the artifact. That distinction is not about caching and it is not about freshness; it is about which of the two systems owns the bytes after the transaction.

Then look at the hand-off itself. Anywhere your pipeline passes something to a system that will not accept a correction — a create-only API, a published package version, an immutable tag, a mail that has been sent — find the last transition before that point and ask what reports its failure. If the answer is nothing, that is where the proof goes, and it is worth paying for even if you never see it fire.