A release is phases, not a button

The appeal of a single release command is obvious. One invocation, everything happens, nothing forgotten. It is also the version that cannot be stopped halfway, and halfway is where you will eventually need to stop.

The case this post is about#

One release, four steps, sorted by whether they can be undone:

build    → produces an artifact            reversible, repeatable, cheap
verify   → checks the artifact             reversible, read-only
stage    → uploads, not yet visible        recoverable, bounded
promote  → reaches users                   NOT reversible

Three of those four have a correct answer and can be retried. The fourth is a judgement about timing that no check can make, and a single release command performs it with exactly the same ceremony as the first one: none.

This post follows those four rows, because collapsing them is appealing for a real reason — one invocation, nothing forgotten — and the thing it removes is the only moment anybody was going to be asked.

The symptom#

The first thing you notice is that the failures are expensive in a way they did not need to be.

A release command that builds, signs, uploads and promotes has failed at step four after doing three and a half minutes of real work. If it is not resumable — and a single command usually is not — the remedy is to run it again from the top. That is tolerable once. On a bad afternoon, where the failure is in the slow external step, you discover you have rebuilt the same artifact four times to retry one upload.

The second shape is worse and quieter: the steps have genuinely different reversibility and the command treats them identically. Building is free and repeatable. Uploading is mostly recoverable. Promoting to users is not — once a release is live, the remedy is another release, and on mobile that means a review cycle. A command that performs all four has made its most dangerous step unceremonious, buried in the middle of a log, with no moment where anybody was asked.

The third is the timing problem, and it is the one that actually forces the design. The decision about when a release reaches users is not a technical question. It is Friday afternoon, or a conference next week, or a partner's launch, or waiting for a fix that is nearly ready. A build is ready when it compiles and passes. A release is ready when somebody says so, and no amount of green checks produces that answer.

The fourth is what a monolithic command does to the operator's attention. When one invocation does everything, the moment of highest consequence has exactly the same ceremony as the moment of lowest consequence — which is to say none. People calibrate care to friction, and a command that makes promotion feel identical to building has removed the signal that would have made somebody think.

Why the obvious fix failed#

Add a confirmation prompt. The reflex fix and it decays fast. A prompt that appears every time, at the same point, with the same wording, becomes a keystroke within a week — and it is asking at the wrong moment anyway, because by the time the command is running the person has already decided to release. A prompt in the middle of a procedure is not a decision point; it is a speed bump placed after the junction.

Make it resumable. This fixes the wasted-work half and leaves the dangerous half exactly as it was. A resumable monolith still promotes to users in the same breath as it compiles; you have made the failure cheaper without making the consequence visible.

A dry run flag. Useful, and it answers a different question — would this work rather than should this go out now. It also has the sharp edge every dry run has: the dry path and the real path are different code, and the thing you verified is not quite the thing that runs.

Schedule it. Push the timing decision into a cron expression. This genuinely works for products where the answer is always "every Tuesday", and it fails for everything else by converting a judgement into a default — the release goes out on Tuesday whether or not Tuesday is still a good idea, and nobody is watching because the whole point was not to have to.

The actual mechanism#

Split on reversibility, and put the human exactly where irreversibility begins.

build    → produces an artifact            reversible, repeatable, cheap
verify   → checks the artifact             reversible, read-only
stage    → uploads, not yet visible        recoverable, bounded
promote  → reaches users                   NOT reversible

Three of those four are mechanical. They have a correct answer, they can be retried, and nothing is lost by running them unattended. The fourth is a judgement that no check can make, and it is the only one that needs a person.

Once the split exists, three things follow that are not obvious beforehand.

Each phase has its own entry condition, so re-running is cheap. A failed upload re-runs the upload, not the build, because the artifact from the previous phase is still there and still valid. That is not an optimisation, it is what makes people willing to retry carefully rather than re-running everything in frustration.

Staging is where verification actually belongs. An artifact uploaded but not promoted is the first point at which you can check the real thing in its real place — the actual binary the store accepted, not a local approximation. Checks that run before upload are checking a different object, and the gap between those two is exactly where upload-time rejections live.

Promotion becomes a separate invocation with its own ceremony, and the friction is the feature. It is a different command, typed deliberately, at a moment chosen for reasons the pipeline does not know. Everything before it can be automatic, scheduled, repeated — because none of it reaches anyone.

The sub-case worth planning for: promotion is often not binary. A staged rollout to a percentage of users is a sequence of promotions, each one a judgement made with information the previous one did not have — crash rates, early reviews, a metric that moved. That makes the phase split more valuable rather than less, because the thing being repeated is the small irreversible step, with a decision before each one, rather than a single all-or-nothing gate.

The fix#

The phases are separate commands, and the state between them is on disk rather than in a process:

release build     → artifact + a recorded manifest of what went into it
release verify    → checks against that artifact
release stage     → uploads it; still invisible to users
release promote   → makes it visible, and only this step is irreversible

Each phase reads the previous phase's artifact off disk, which is what makes a resume exact rather than a re-run:

phase_build() {
  build_artifact > "$REL/app.aab"
  # The manifest is what later phases verify AGAINST. Recording it here, from the
  # build that just happened, is the difference between verifying the artifact and
  # verifying your assumptions about it.
  cat > "$REL/manifest.json" <<JSON
{ "artifact": "app.aab",
  "sha256": "$(hash_file "$REL/app.aab")",
  "version": "$(read_version)",
  "built_at": "$(iso_now)" }
JSON
}

phase_verify() {
  want=$(json_get "$REL/manifest.json" sha256)
  got=$(hash_file "$REL/app.aab")
  [ "$want" = "$got" ] || { echo "artifact changed since build: $want != $got" >&2; exit 3; }
  run_release_checks "$REL/app.aab"
}

phase_stage()   { upload_to_track internal "$REL/app.aab"; record_phase staged; }
phase_promote() { require_phase staged; promote_to production; record_phase promoted; }

Resuming picks up from recorded state rather than from what somebody remembers doing:

$ release verify
  artifact app.aab  sha 0c57da24  ok
  checks 14/14 pass

$ release promote
error: phase 'staged' not recorded — run `release stage` first

That refusal is what earns the separation. Promote is the only irreversible step, and it is gated on a recorded fact rather than on the operator having done things in the right order.

What makes this work in practice is that the state between phases is inspectable. After staging, you can see what was staged — the version, the artifact hash, what it was built from — without re-deriving any of it. A pipeline whose intermediate state only exists inside a running process forces you to either trust it or start again.

The entry conditions are worth stating explicitly rather than leaving implied, because they are what stops the phases becoming four commands you have to remember the order of. Each one declares what it requires and refuses without it: verify needs an artifact, stage needs a verified one, promote needs something staged. Run them out of order and the refusal tells you which phase you actually wanted, which means the sequence is discoverable from the tooling rather than from a document.

The rollback path deserves the same treatment and usually does not get it. If promotion is a separate command, so is its reverse, and the reverse needs to be rehearsed rather than written down — because the day you need it is the day nobody is calm.

There is a failure mode the split introduces and it is worth designing against rather than discovering: phases that drift apart. If promote can run against whatever happens to be staged, then staging something new and promoting something old are two commands that will eventually be run in the wrong order by somebody who assumed the obvious. The artifact identity has to travel with the phase — promote names what it is promoting, and refuses if that is not what is actually there.

That check costs almost nothing and it converts the split's one genuine weakness into a refusal. It also makes the output useful: a promote that states which artifact, built from which commit, at which version, is a line worth keeping, and it is exactly the line you want when somebody asks three weeks later what went out.

How we proved it#

The phases exist as separate commands and each one can be run on its own, with the artifact and its manifest persisting between them. That is the structural claim and it is checkable by running them out of order and watching the entry conditions refuse.

The evidence I find most convincing is about retries. With the phases split, a failed upload costs one upload. Before, the same failure cost a full rebuild, and the rebuild was not incidental — it was several minutes each time, which is what turned a small external flake into an afternoon.

What I cannot claim is a prevented bad release. The argument for putting the human at the promote step is that it preserves a moment to say no, and I cannot show you a case where I used it. The honest version is that this is a design that keeps an option open, and the value of an unexercised option is not something I can evidence — only argue for.

What I have not solved: the rehearsed rollback. I said above that the reverse path needs practising rather than documenting, and mine is documented. I have not run it under conditions resembling a real incident, which means I know the command exists and I do not know that I can execute it under pressure with the store's review timing working against me.

And a limitation of the split itself: more commands is more surface to get wrong, and the failure it introduces is a release that stops halfway and is forgotten — staged, never promoted, sitting there looking finished. A monolith at least has the virtue that it either happened or it did not. Nothing in my setup notices a build that has been staged for three weeks, and that is the cost of the design rather than an oversight in it.

What this generalizes to#

When a procedure mixes reversible and irreversible steps, the useful seam is where recoverability changes, not where the code happens to be organised.

Most pipelines are split by technical concern — compile, package, deploy — because that is how the work decomposes. Splitting by reversibility cuts across that and produces something more useful: the phases before the seam can be automated, retried and scheduled without thought, and the phase after it gets a person, a separate invocation and whatever ceremony its blast radius deserves.

Then look for the step that needs a judgement no check can make. There is usually exactly one, it is usually about timing rather than correctness, and it is usually buried in the middle of something automatic. Pulling it out is most of the work; what is left on either side of it is simpler than it was.