Signing state has exactly one owner, or it has none

Signing is not one piece of state. It is a certificate, a private key, a provisioning profile, an entitlements file, a team id, and the machine-level question of which keychain any of that lives in.

Every one of those can be set by more than one thing. That is the entire problem.

The case this post is about#

One keychain, with two things asserting what is in it:

second writer   a human, once, as cleanup
first writer    the provisioning flow, on a schedule
disagreement    surfaced at the next build, not at the change
time to find    most of a day, in the wrong repository

The flow installs certificates and asserts the keychain's state every time it runs. A person tidied up a leftover temporary keychain, once, as a reasonable piece of housekeeping.

Nothing conflicted at the moment of the change. The disagreement appeared at the next build as identity not found, which reads as a certificate problem and sent the investigation to the certificate repository, where nothing had changed.

This post follows those four lines, because competence was never the issue — two writers to one piece of state is the whole bug.

The symptom#

A build signs on one machine and not another. Or it signed last week and does not today, with no commit in between that touches anything related.

The investigation is unpleasant because each piece looks fine in isolation. The certificate exists. The profile is present and not expired. The team id is right. Nothing is obviously wrong, and yet the combination does not resolve — because the question is never "is this piece valid", it is "which piece won", and nothing records that.

What makes it worse is that the failure is delayed. A second writer touching signing state causes no error at the moment it happens. It causes an error at the next build, which might be hours later, on a different machine, run by someone who did not make the change.

Why the obvious fix failed#

The obvious fix is documentation: write down how signing is set up, and have everyone follow it.

That fails for the reason all unenforced convention fails, with one aggravating factor specific to this domain — signing breaks at the worst possible moment, usually a release, and a documented process is exactly what people abandon under time pressure. The one-line manual fix is always available and always looks faster than running the flow.

The second attempt is to automate the common path and leave manual escape hatches for the rest. That is better and it is where most teams land. The trouble is that an escape hatch used twice becomes the path, and now two things assert the same state with no record of which ran last.

The actual mechanism#

One flow asserts all of it, idempotently, and nothing else writes.

Idempotent, not conditional. The flow should be safe to run at any time and should converge the machine to the correct state regardless of what it finds. A flow you can only run "when things are broken" is one nobody runs until things are broken, which means it is never exercised and is itself probably broken.

It owns the keychain, not just the certificates. Which keychain is default and what is in the search list are part of signing state, not environment trivia. If the flow does not assert those, it has left a gap that something else will fill.

Fetching is not installing. A flow that downloads material and leaves placing it to a human has not removed the second writer; it has renamed it.

The build tool's temporary keychain is out of scope. Fastlane creates and destroys its own per build. That lifecycle belongs to the build, and the provisioning flow should neither create nor remove it.

The fix#

The structural move: make the flow the only thing with write access, then make running it cheap.

Those two halves are one decision. A flow that is the sole writer but takes ten minutes and prompts for a password will be routed around on a bad afternoon. The reason it can be the sole writer is that it is faster than the manual alternative, not that a rule says so.

The flow asserts the state it needs rather than assuming it, and repairs what it can — which is what makes "just re-run it" a real answer instead of a deflection:

preflight() {
  # 1. HYGIENE first. Default keychain and search list drift for ordinary reasons
  #    (a crashed build, a tmp keychain that outlived its run). Repair, do not fail.
  ensure_default_keychain "$LOGIN_KEYCHAIN"
  ensure_in_search_list   "$LOGIN_KEYCHAIN"

  # 2. Is there a usable identity? Ask the system, not a config file.
  if identity_present "$TEAM_ID"; then
    echo "signing identity present"
  else
    echo "no identity for $TEAM_ID — fetching"
    fetch_signing_material            # the one writer
  fi

  # 3. Prove it, by the same question the build will ask.
  identity_present "$TEAM_ID" || {
    echo "preflight ran and the identity is still absent — not a keychain problem" >&2
    exit 5
  }
}

On a machine whose default keychain had been cleared:

$ signing-preflight
  default keychain was unset — restored to login.keychain-db
  login.keychain-db was not in the search list — added
  no identity for ABCDE12345 — fetching
  signing identity present

Three repairs and a fetch, in eleven seconds, with no decisions asked of the operator. That is the half that earns sole-writer status: the manual alternative is slower and more dangerous, so there is nothing to route around.

The hygiene step is the part I would argue for hardest, because it looks like scope creep. Repairing the default keychain has nothing to do with fetching certificates, and putting it in the same command feels like a tidy-up bolted onto a fetch. It belongs there for a specific reason: keychain drift is the single most common cause of "signing broke and nothing changed", and it is the cause whose error message points somewhere else entirely. If the flow does not repair it, the person debugging will go looking in the certificate repository, and the certificate repository will be fine.

That is the general shape of making a flow the sole writer: it has to absorb the adjacent failures too, or people will reach past it for the adjacent thing. A fetch-only command leaves keychain problems to manual surgery, and manual surgery is exactly what sole-writer status was meant to eliminate.

The second property is that every assertion asks the system the same question the build will ask. Checking a config file, or a path, or a recorded fact about a previous run is checking a proxy — and the proxy can be right while the thing is wrong. Asking for the identity by team id is slower by milliseconds and is the only question whose answer means anything.

What this does not cover is the case where the fetch itself is the problem — a revoked certificate, a hit device limit, an expired membership. The flow reports those honestly rather than repairing them, because they are not local state and no amount of local repair touches them. Exit 5 above is deliberately phrased to say so: the preflight ran, the identity is still absent, and the next place to look is not this machine.

There is a cost to this design that is worth stating. Concentrating write access in one flow means that flow is now load-bearing for every developer on every machine, and a bug in it is a bug everybody has at once. I have traded a diffuse risk — anyone can break their own keychain — for a concentrated one, and the trade is only correct because the diffuse version was happening regularly and the concentrated one has not yet happened at all.

The mitigation is that the flow is small and does one thing. It fetches signing material and asserts keychain hygiene, and it has no opinion about builds, versions, or anything else. A sole writer that accumulates responsibilities becomes a sole point of failure for all of them, and the discipline that keeps this one safe is refusing to put the next convenient thing inside it.

Concretely that means it runs non-interactively, it repairs rather than refuses when it finds drift, and it reports what it changed. The last one matters more than it sounds: a flow that silently converges is indistinguishable from a flow that did nothing, and the first time you need to know which, you will be mid-incident.

What remains for a person is deciding when — before a release, after a certificate rotation, when onboarding a machine. Not what.

There is a second-order benefit that only shows up once the flow is genuinely the only writer: the machine becomes reproducible. Not in the container sense — an unplugged laptop with a login keychain is never that — but in the sense that any two machines which have run the flow agree, and any machine that has not is identifiably in a different state rather than mysteriously different. That turns "works on mine" from an argument into a question with an answer.

It also changes what a new machine costs. Onboarding stops being a document somebody follows with varying fidelity and becomes one command whose failure modes are the flow's failure modes, already known. The first time that matters is the first time somebody joins mid-release.

How we proved it#

The evidence is a failure, and it is the cleanest argument for single ownership I have.

A temporary keychain left behind by a crashed build was removed by hand as cleanup. It happened to be the default keychain, so removing it left the machine with no default at all and no signing identity resolvable anywhere. The investigation spent most of a day inside the certificate repository, which had not changed and was never at fault.

second writer   a human, once, as cleanup
first writer    the provisioning flow, on a schedule
disagreement    surfaced at the next build, not at the change
time to find    most of a day, in the wrong repository

What changed after: the flow gained a keychain hygiene step that asserts the default and the search list every run, and the manual commands that could contradict it are refused outright. The second writer was not asked to be careful. It was removed.

Also measured, and the part I did not expect: making the flow assert keychain state fixed a class of "works on my machine" failures nobody had connected to signing. Machines drift for ordinary reasons — an unrelated tool creating a keychain, an OS update, a restored backup — and once one thing reliably converges that state, the drift stops being anybody's problem.

What I have not proven: that this survives a team large enough to have people who never read the rule. Everything above is from a small number of machines where the sole-writer property could be enforced mechanically. At a scale where someone will run the manual command regardless, the guard has to live somewhere they cannot route around — CI, or the machine image — and I have not operated it at that size.

The honest boundary is narrower still: the guard refuses commands at the shell an agent drives. A person typing into their own terminal is outside it. What makes that acceptable here is that the flow is faster than the manual path, so the incentive and the rule point the same direction — and I would not trust the rule alone if they did not.

What this generalizes to#

For any piece of state that more than one thing can set, the question is not how to coordinate them. It is which one owns it, and what you have to do to the others so they cannot write.

Then make the owner cheap to run. Sole ownership enforced by a rule lasts until the first bad afternoon; sole ownership that is also the fastest path survives it. And have the owner say what it changed, because a convergence you cannot see is one you will eventually have to take on faith at exactly the wrong moment.