The submodule pointer you accidentally reverted

An outer repository records a submodule as a single commit hash. Staging everything in that outer repository stages whatever hash the inner repository currently sits at.

Most of the time that is the newest one, and nothing interesting happens. Sometimes it is older than the hash already committed, and then you have silently published a rollback.

The case this post is about#

One line, in a commit touching forty files:

+Subproject commit 2b84ef0…

That is a submodule pointer moving. Nothing in it says whether it moved forward to work that was just merged or backward to a checkout from last week, and both look identical — one line, two abbreviated hashes, no direction.

This one went backward. A deliberate advance to a merged commit was reverted by a routine git add -A in the parent, which stages whatever the working tree's submodule currently has checked out, and the commit message was about something else entirely.

This post follows that one line, because a reviewer reading a forty-file diff is not going to resolve two hashes against a remote, and the information needed to tell the two cases apart is not in the diff.

The symptom#

What lands in the commit is a two-line diff:

-Subproject commit 7f3a9c1…
+Subproject commit 2b84ef0…

That is the entire record of the change, and read it as carefully as you like — it does not say which of those two is newer. A commit hash carries no date you can see, no sequence number, no ordering of any kind at a glance. Two hashes are two opaque strings, and the diff format presents a rollback and a legitimate advance in exactly the same shape.

The second thing is where that diff sits. A bottom-up commit across three repositories touches real work — source files, config, documentation. The pointer line arrives in the middle of forty other files, as the smallest change in the set, in a commit whose message is about something else entirely. It is the single most skimmable line in the diff and the only one that can undo a day's work.

The third is who notices and when. Not you: your working tree is fine, because the inner repository on your disk is at whatever state you were using. The person who notices is whoever pulls next and runs a submodule update, and what they experience is their own inner repository moving backwards — code they had, gone, with no commit in the inner repository explaining it, because the change was not made there.

The fourth is that it recurs. Fix the pointer, push, and the next time you commit from a checkout whose inner repository is behind, the same thing happens again. It is not a mistake someone made once; it is a standing property of staging everything in a tree that contains a pointer to somewhere else.

The sharpest version of that is a merge. Someone merges the inner repository's pull request and advances the outer pointer to the merged commit — a deliberate, correct, reviewed act. Your checkout still has the inner repository at its pre-merge state because you have not updated it. You commit something unrelated in the outer repository, stage everything, and the pointer goes back to before the merge.

Nothing about that sequence contains a mistake you could point at. The merge was right. Your commit was about something else. The two are connected only through a hash you never looked at, and the result is that a merged change is no longer the one the project points to — which is a far stranger state than an unmerged one, because the pull request is closed and everything in the inner repository says the work landed.

Why the obvious fix failed#

Stop staging everything. The direct answer, and it is barred here for a reason that took its own incident to learn. A bottom-up commit is meant to be unconditional: stage everything in each repository, commit, move up. The moment you start choosing which paths to include, you are making attribution decisions mid-commit — does this file belong to this session, this feature, this workspace — and that is a pause with no good stopping point. Selective staging buys pointer safety with a problem that is worse and far more frequent.

Update the submodule before every commit. Reliable and expensive: a network round trip on every commit, which turns a two-second operation into a ten-second one and trains people to skip it. It is also wrong whenever the inner repository is deliberately behind — a pinned version, a reverted dependency, a branch you are not ready to adopt. "Always take the newest" is a different policy, not a safety check.

Review the diff. The one everybody believes in and the one the format defeats. You cannot order two hashes by looking at them. A reviewer who genuinely wants to check this has to go to the inner repository and ask about ancestry — which is a real command, run by hand, for one line, in a forty-file diff. Nobody does this, and the reason is not carelessness.

Track a branch instead of pinning a commit. This makes the problem disappear by removing the feature. Submodules pin to a commit on purpose: it is what makes a checkout reproducible. Trading that away to avoid a staging hazard gives up the thing the mechanism exists for.

The actual mechanism#

The missing ingredient is ordering, and git will supply it on request.

Two hashes are unordered. Two hashes plus the inner repository's history are not: one of them is either an ancestor of the other or it is not, and that is a question with a definite mechanical answer. git merge-base --is-ancestor <staged> <committed> returns true exactly when the thing you are about to stage is older than the thing already recorded — which is the definition of a rollback.

That turns an unanswerable review question into a boolean, and it is the whole trick. Everything else about the diff stays exactly as unhelpful as it was; what changed is that the check no longer depends on a human recognising a hash.

Three properties make the guard usable rather than annoying.

It is surgical. When a rollback is detected, the fix is to unstage that one path and leave its committed value alone. Everything else in the commit proceeds untouched. The temptation is to abort the whole commit and ask — and aborting is precisely the wrong move, because it converts a question with a known correct answer into a human interruption, which is the thing the unconditional-staging rule exists to avoid.

It is the only exception. The surrounding discipline is absolute: stage everything, commit, do not deliberate. A single carved-out exception survives in a rule like that; a second one starts a negotiation, and within a month the rule has conditions and nobody can state it.

The ordering of repositories still matters. Inner repository first, then the outer pointer. The guard protects against a backwards pointer; it does not help with a pointer that is ahead of anything pushed, which is what you get if you commit the outer repository first. Those are two different failures and they need both halves — the ancestry check and the bottom-up order.

There is a case the ancestry test deliberately does not catch, and it is worth being clear about it. If the staged hash is neither an ancestor nor a descendant — a genuinely divergent branch — the check reports no rollback, because there is none. That is correct, and it is also the case where a human should probably look. The guard answers "is this backwards", not "is this right", and conflating the two would give it a confidence it has not earned.

The fix#

The guard runs between staging and committing, in the outer repository:

for each staged submodule path
    staged    = the hash about to be committed
    committed = the hash currently in HEAD
    if staged is an ancestor of committed
        unstage that path, keep the committed value, say so

Four lines of behaviour, and the reason it is worth writing down is the last one. A guard that silently corrects produces commits that do not match what the operator thought they were doing, which is its own small betrayal. Naming the path and the two hashes costs one line of output and means the rollback is a thing that was noticed rather than a thing that was prevented invisibly.

It also belongs in the commit path rather than in a pre-commit hook on the repository, for a slightly subtle reason: hooks are local, uninstalled by default, and absent on a fresh clone. A check that lives in the command everybody actually runs is present the first time somebody runs it.

The implementation is the ancestry test, and that is the whole of the judgement — a pointer moving backwards is a regression, a pointer moving forwards is the normal case:

guard_submodule_pointers() {
  git diff --cached --name-only | while read -r path; do
    [ -f "$path/.git" ] || [ -d "$path/.git" ] || continue      # not a submodule

    staged=$(git diff --cached --submodule=short -- "$path" | awk "/^\+Subproject/{print \$3}")
    committed=$(git ls-tree HEAD -- "$path" | awk "{print \$3}")
    [ -n "$staged" ] && [ -n "$committed" ] || continue
    [ "$staged" = "$committed" ] && continue

    # The test: is the value we are about to commit an ANCESTOR of the one already
    # committed? If so this is a rollback — almost always `add -A` re-staging an
    # older checkout — and the committed value is the one to keep.
    if git -C "$path" merge-base --is-ancestor "$staged" "$committed" 2>/dev/null; then
      git restore --staged "$path"
      echo "kept $path at $committed (refused rollback to $staged)"
    fi
  done
}

On a tree where one submodule had been updated and another was stale:

$ commit
  kept source/app at 9f2c1b05 (refused rollback to 243a9b3d)
  staged 14 files, 1 submodule pointer advanced

The printed line is the part that matters as much as the restore. A silent correction produces a commit that does not match what the operator thought they were making.

How we proved it#

The strongest evidence I have is structural rather than statistical, and I would rather say that plainly than dress it up.

The commit discipline here is deliberately absolute — stage everything, in every repository, bottom up, with no attribution questions and no selective exclusions. That rule was written after selective staging cost real time, and it is stated in the strongest terms available precisely so that it is not negotiated with.

This guard is the only exception in it. A rule written to be unconditional, which nonetheless carves out one mechanical carve-out for submodule pointer rollbacks, is a rule that met this failure and decided it was worth the cost of an exception. That is the measurement: not a count, but the shape of what the discipline was willing to give up.

What I have not measured: how often it fires. I have no counter on it and no log of prevented rollbacks, so I cannot tell you whether this guard has saved me ten times or zero. That is a real gap — a guard with no telemetry is a guard you cannot evaluate, and "it has never caused a problem" is equally consistent with "it works" and "it never runs".

And a second honest limit: the ancestry test is only as good as the inner repository's local history. If the committed hash is not present locally — a shallow clone, or a commit that was force- pushed away — merge-base cannot answer and the check has to decide what to do with an unknown. The safe choice is to treat unknown as "do not touch the pointer", which means on a shallow checkout the guard quietly degrades to doing nothing. It does not fail loudly, and it probably should.

What this generalizes to#

Any time your system records a reference to a version rather than the version itself, you have inherited this problem: references can move in two directions and most diff formats show both the same way.

Lockfiles, pinned image tags, vendored dependency manifests, pointer files in a data store — all of them produce a change that looks identical whether it is an upgrade or a downgrade, and all of them sit in diffs where that one line is the least interesting thing on screen.

The move is the same every time. Find the system that does know the ordering — the package registry, the image history, the inner repository's commit graph — and ask it, mechanically, at the moment the change is staged. Ordering is almost always available somewhere; it is just never in the diff, which is the one place everybody is looking.