Three repositories, one change, and the order that stops it corrupting

One logical change touches three repositories: a source project, the workspace that contains it, and the framework that contains that. Each is a submodule of the one above.

There is exactly one order in which committing them produces something another machine can check out.

The case this post is about#

One logical change across three nested repositories, and the only order that produces something another machine can check out:

source      commit, push            the commit now exists on the remote
workspace   commit, push            its pointer refers to a published commit
framework   commit, push            same, one level up

Reverse it and nothing fails locally. The workspace commits a pointer to a source commit that exists only on the machine that made it, the build works, the tests pass, and the repository is internally consistent on that disk.

Somebody else clones it and submodule initialisation fails on a commit that does not exist anywhere they can fetch from.

This post follows those three lines, because the defect is entirely about what is reachable from somewhere else, and the person who caused it is never the person who finds it.

The symptom#

A colleague clones the repository, initialises the submodules, and one of them fails to fetch. The commit the parent points at does not exist on the remote — it exists only on the machine where the work was done, because the parent was committed and pushed before the child was.

The second thing is that nothing was wrong locally. The person who did it ran a perfectly ordinary sequence, the build worked, the tests passed, and the repository was internally consistent on their disk. The pointer is a reference to a commit, and on their machine the commit is right there. The defect is entirely about what is reachable from somewhere else.

The third is the reverted pointer, which is the quiet one. Somebody advances a submodule pointer deliberately — to a merged commit, say — and a later git add -A in the parent, run by automation or by habit, stages the pointer at whatever the working tree's submodule currently has checked out. If that is the older commit, the deliberate advance is reverted, in a commit whose message is about something else entirely.

The fourth is that the diff makes it almost impossible to see. A submodule pointer change shows as one line with two abbreviated hashes, and a reverted pointer looks exactly like an advanced one. Nothing about the presentation distinguishes "moved forward to the thing we merged" from "moved back to where it was last week", and a reviewer reading a twelve-file diff is not going to resolve two hashes by hand.

Why the obvious fix failed#

Commit everything at once with a script that walks the repositories. Right instinct and it fails if the walk is in the wrong order, which is the whole question. A script that iterates in directory order or alphabetically will get it right some of the time, which is worse than always getting it wrong because the failures look random.

Push the parent and fix it afterwards. What usually happens, and it works in the sense that the state converges. It also means the broken pointer existed on the remote for some window, and anybody who cloned or pulled during that window got the failure — which is how this is usually discovered, by somebody else.

Avoid submodules. Entirely reasonable and not available to everybody. The structure exists because the repositories have different owners, lifecycles and access, and the pointer problem is the cost of that structure rather than a mistake in adopting it.

The alternatives have the same problem wearing different clothes, which is worth knowing before switching. A monorepo removes it by removing the boundary. Package versions replace an address with a name and move the failure to publication time, where it is louder and still real — a manifest naming a version that was never published fails in exactly the same way, at a different moment.

Review the pointer changes carefully. Asks a human to resolve abbreviated hashes against remote branches during a code review. It is the kind of check that works once, when somebody has been burnt recently, and does not survive contact with an ordinary week.

The actual mechanism#

Commit and push bottom-up — the innermost repository first, then its parent, then the parent's parent — so that every pointer refers to something already published.

source      commit, push            the commit now exists on the remote
workspace   commit, push            its pointer refers to a published commit
framework   commit, push            same, one level up

The rule is a consequence of what a pointer is: a reference, stored in the parent, to a commit in the child. A reference is only resolvable if the thing it refers to is reachable from where the reference lands. Committing the parent first stores a reference to something nobody else can fetch — the pointer is not broken on your disk, which is exactly why you do not notice.

The sub-case that has to be mechanical rather than remembered is the reverted pointer. When staging everything in a parent repository, a submodule pointer that was deliberately set to a newer commit must not be staged back down to whatever the working tree has. That cannot be a rule people follow, because it fires during a bulk operation that is otherwise safe, and the person running it is thinking about something else. It has to be a check in whatever performs the commit: compare the pointer being staged with the one already recorded, and refuse to move it backwards without an explicit instruction.

There is a related ordering question on the way back that people get wrong in the opposite direction. When updating — pulling other people's work — the correct order is top-down: take the parent's view of where the children should be, then update the children to match. Commit bottom-up, update top-down, and the asymmetry confuses everybody including me until you notice that both directions are just "publish before you refer, read the reference before you follow it".

The fix#

commit    innermost first, push before the parent is committed
refuse    a pointer that moves backwards, unless explicitly instructed
update    top-down, which is the opposite direction and correct
automate  all of it, because the order is not a thing to remember

Innermost first, pushed before the parent records it — because a parent committing a pointer the remote has never seen is a reference to nothing for everybody except the machine that made it:

commit_all() {
  local msg="$1"

  # BOTTOM-UP. Each level pushes before the level above records its hash.
  for repo in "$SOURCE" "$WORKSPACE" "$FRAMEWORK"; do
    stage_all "$repo"

    # A pointer that moves BACKWARDS is a re-stage of an older checkout. Keep the
    # recorded value; everything else in the tree still goes in.
    for sub in $(submodules_of "$repo"); do
      staged=$(staged_hash "$repo" "$sub")
      recorded=$(recorded_hash "$repo" "$sub")
      [ -n "$staged" ] && [ "$staged" != "$recorded" ] || continue
      if is_ancestor "$repo/$sub" "$staged" "$recorded"; then
        unstage "$repo" "$sub"
        echo "  kept $sub at $recorded (refused rollback)"
      fi
    done

    has_staged_changes "$repo" || { echo "  $repo: nothing to commit"; continue; }
    record_commit "$repo" "$msg"
    publish "$repo"                     # BEFORE the parent records the pointer
    echo "  $repo: committed + pushed"
  done
}

A run where the innermost repository had a stale pointer staged:

$ commit "fix: worked examples"
  source/app: kept vendor/sdk at 9f2c1b05 (refused rollback)
  source/app: committed + pushed
  workspace:  committed + pushed
  framework:  nothing to commit

Reverse the order and the workspace commits a pointer to a source commit that exists only locally. It resolves on the machine that made it and is a dangling reference everywhere else — which is why the failure is always reported by somebody other than the person who caused it.

The fourth line is the one that actually fixes it. Everything above can be done by hand correctly by somebody who has read this and is paying attention, and that describes the situation that produced every incident in this post. The order is a property of the structure, it never varies, and it is therefore exactly the kind of thing that should be in a script rather than in a convention.

What makes that script small is that it is doing nothing clever: enumerate the repositories from the inside out, commit and push each, and check the pointer direction before staging. The whole value is that it cannot forget and cannot be in a hurry.

There is one decision inside it that is not mechanical, and it is worth making deliberately: what the script does when the working tree is dirty in a repository it is about to commit. The tempting answer is to stash, and it is wrong — stashing hides somebody's work in a place they will not look for it, and the recovery is manual and unobvious. Committing the dirty state to the current branch keeps it visible and keeps it theirs, which is the property that matters when the alternative is silent loss.

How we proved it#

The failure is reproducible in two commands. Commit a parent whose submodule pointer refers to an unpushed commit, then clone it somewhere else. The clone fails at submodule initialisation, with a message about a commit that does not exist, and the demonstration takes under a minute.

The reverted-pointer case cost about three hours and is the reason the direction check exists. A bulk staging operation in a parent repository reverted a pointer that had been deliberately advanced, the commit was pushed, and the work that pointer referred to became unreachable from the parent — not lost, since the commits existed, but absent from every checkout and not obviously so.

What I cannot tell you is how often the broken-pointer case had happened before I noticed. It is only visible to somebody cloning or updating in a particular window, and those people reasonably assume a transient problem and try again later — by which time the child has been pushed and it works. A failure mode that self-heals is a failure mode nobody reports.

That is worth generalising for a second, because it explains why this survived as long as it did. A defect whose symptom disappears if you wait produces no bug report, no incident, and no pressure to fix it — it produces a small shared belief that the tooling is occasionally flaky. Several people can hold that belief for a year without ever comparing notes, and the thing they are describing is perfectly deterministic.

What I have not solved: partial failure. If the innermost push succeeds and the next one fails — network, permissions, a protected branch — you are left in a state that is half published, and the recovery is manual. The script makes the common path correct and does not make the interrupted path safe, and I have not decided whether a real transaction here is worth the complexity.

And a limit worth being honest about: none of this makes submodules pleasant. It makes one specific class of corruption impossible, and everything else about the structure — the detached heads, the confusing status output, the two-step update — remains exactly as awkward as it was. I would not adopt submodules because the ordering is solvable; I would solve the ordering because the submodules are already there.

The thing that made it click was stopping thinking of a pointer as a version number. A version number is a name, and a name can be written before the thing it names exists. A submodule pointer is an address, and an address that refers to somewhere unreachable is not a slightly stale address — it is simply not resolvable, and the whole ordering rule falls out of that one distinction.

What this generalizes to#

Publish before you refer to it, and the order follows from which artifact contains the reference.

The same constraint appears wherever one artifact names another across a boundary. A deployment manifest naming an image tag that has not been pushed. A release note linking a commit that is still local. A document referencing a page that is not yet live. A migration recording a version the schema repository has not published. In each case the reference is valid from where it was written and invalid from everywhere else, which is the property that makes it hard to notice.

So the useful question when a change spans several artifacts is which one contains the reference and which one contains the referent — because the referent has to be reachable first, every time, and the direction of the dependency is the thing that determines the order rather than any intuition about which part is more important.