The keychain repair that broke signing for a day

A build failed to sign. In the logs was a temporary keychain with a build-id suffix, left behind by a previous run. Obvious debris. I removed it.

It was the default keychain. Nothing else on the machine could find a signing identity for the rest of the day, and the repository I spent that day suspecting had never been wrong.

The case this post is about#

One command, run against what looked like debris from a crashed build:

observed   a tmp keychain with a stale build id
action     removed it by hand
result     default keychain unset · no signing identity found, machine-wide
suspected  the provisioning repository (wrong — it had not changed)
offered    "Reset To Defaults" — a bigger change than the original mistake

The keychain was named for a build that had finished days earlier. It was also the default keychain, so removing it left the machine with no default and nothing able to find a signing identity.

The error that followed says the identity could not be found, which reads as a certificate problem. This post follows those five lines, because the diagnosis pointed at the wrong repository for most of a day and the thing that would have prevented it is not expertise.

The symptom#

Signing failures do not say "your keychain search list is broken". They say the identity could not be found, which reads as a certificate problem, which sends you to the certificate repository.

So the search starts in the wrong place. You re-run the provisioning flow. You check whether the certificate expired. You look at the profile's embedded entitlements. Every one of those is a reasonable hypothesis and every one is downstream of a machine-level setting nobody thought to check, because nothing in the error mentions it.

The tell, once you know to look for it, is that everything fails — not one target, not one configuration. A certificate problem is usually narrow. A keychain problem is total.

That gap between what an error names and where the fault lives is the whole cost here. An identity lookup failing is the last step in a chain — the certificate has to exist, in a keychain, that is in the search list, one of which is the default. The message reports the end of the chain, so the investigation starts at the end too, and works backwards through the expensive hypotheses first.

Why the obvious fix failed#

The obvious fix is to clean up the mess. A keychain named for a build that finished days ago is clearly garbage, and the command to remove one is right there.

What that misses is that macOS tracks two things separately from whether a keychain is useful: which keychain is the default, and which keychains are in the search list. A temporary keychain created by a build tool can end up as either. Removing it does not hand those roles back to the login keychain — it leaves the default unset, and the search list short.

The second attempt is worse, and the system offers it to you. When something asks for a keychain that is gone, macOS shows a dialog with a Reset To Defaults button. It looks like exactly the repair you want. It is a much larger change than the one you were trying to make, and it is the reason a one-command mistake became a day.

The actual mechanism#

Keychain state has an owner, and it is not you at a prompt.

The provisioning flow owns the certificates. Fetching, installing and renewing signing material is one path, and it is the only thing that should be putting identities on the machine.

The build tool owns its own temporary keychain. Fastlane creates one per build and tears it down. Its lifecycle is not yours to manage, and a leftover one is evidence of a crashed build, not an invitation.

The repair is the flow, not surgery. When signing breaks, the correct action is to re-run the provisioning preflight — which also repairs the default keychain and the search list, because those drift for ordinary reasons and the flow is where that knowledge lives.

What that leaves for a human to type is: nothing. Read-only inspection is fine — listing identities, listing keychains, showing a keychain's settings. Every mutating command is the flow's business.

The split is deliberately drawn between reading and changing, not between safe and dangerous. Every command on the left tells you what the machine currently believes and alters nothing, which makes them the right first move when signing breaks. Everything on the right alters which keychain is default, what is in the search list, or what material is installed — and those three are precisely what the flow asserts on every run.

The fix#

The rule I would write, and did: no ad-hoc keychain mutation, ever.

Specifically refused: removing a keychain or a certificate, setting the default, setting the search list, creating a keychain, changing its settings, and importing signing material by hand. Those are the verbs that change machine state outside the flow that is supposed to own it.

The enforcement is a hook that refuses the mutating verbs and leaves the read-only ones alone. That distinction is the whole design: an agent that cannot inspect the keychain will improvise around the guard, so inspection has to stay free.

# PreToolUse hook. Exit 2 blocks the command.
cmd="$1"

# Read-only inspection is always allowed. Blocking it is how you get a workaround.
case "$cmd" in
  *"find-identity"*|*"find-certificate"*|*"list-keychains"*|*"show-keychain-info"*)
    exit 0 ;;
esac

# The mutating verbs. Note `default-keychain -s` MUTATES while `default-keychain`
# alone reports — the two are one flag apart, so the pattern must be precise.
if printf '%s' "$cmd" | grep -qE \
     'keychain-tool +(remove|remove-cert|create|import|configure)|keychain-tool +(default|search-list) +[^|]*-s'
then
  cat >&2 <<'MSG'
refusing: ad-hoc keychain mutation.
  Signing keychain state is owned by the provisioning flow, not by hand.
  cert or profile problem   -> re-run the signing preflight
  default/search-list drift -> the preflight's hygiene step repairs it
  genuinely stuck           -> a HUMAN runs it in a real shell with the bypass
MSG
  exit 2
fi
exit 0

The one-flag distinction is the part worth getting right:

$ keychain-tool default
    "/Users/me/Library/Keychains/login.keychain-db"     # allowed — reports

$ keychain-tool default -s /tmp/build.keychain
refusing: ad-hoc keychain mutation.
  Signing keychain state is owned by the provisioning flow, not by hand.
  ...
$ echo $?
2

What I actually ran was a remove against a leftover temporary keychain that happened to be the default. It cleared the default, and the next build could not find any signing identity at all — which is a much larger failure than the tidy-up was worth.

The detail that makes this guard survivable rather than infuriating is the escape, and where it lives. There is no agent-usable flag — the bypass requires a human to export it in a real shell, so it cannot be reached by the thing being guarded. That sounds absolute and it is deliberately asymmetric: the cost of a blocked legitimate operation is a human typing one command, and the cost of an unblocked illegitimate one is a machine that cannot sign anything until somebody who understands keychains sits down with it.

The second detail is that the guard's message names the sanctioned path rather than just refusing. A refusal with no route produces a workaround — somebody disables the hook, or runs the command from a different shell, and the rule is gone with nothing recording that it went. A refusal that says "re-run the preflight, it repairs this non-interactively" is an instruction, and the instruction is usually faster than the surgery would have been.

What this does not protect against is the preflight itself being wrong. The whole design concentrates authority in one flow, which is the right trade for state this fragile, and it means a bug there has no second opinion anywhere. I would rather debug one owner than reconcile three.

A rule like that is only worth writing if something enforces it, because the moment signing is broken is exactly the moment a plausible one-line repair is most tempting. So it is a pre-execution guard that refuses the command rather than a line in a document. A genuinely stuck machine needs a human to override it deliberately — which is the right bar, because a human overriding it is a decision, and an agent reaching for it mid-incident is a reflex.

There is one more thing I would tell anybody standing in front of that dialog: the Reset To Defaults button is not the repair it appears to be. It is offered at the exact moment you are looking for a quick way out, it is phrased as a restoration, and what it actually does is wider than the mistake you were trying to undo. The correct move from that screen is to close it and re-run the provisioning flow, which repairs the specific thing that broke and leaves everything else alone.

The general shape, past keychains: when a system offers you a repair at the moment something is broken, check whether the repair is scoped to the break. The ones that are phrased most reassuringly are often the widest.

The distinction the guard draws is between reading and changing, not between safe and dangerous commands. Listing identities tells you what the machine believes and changes nothing, so it stays available and is usually the first useful thing to run. The refusal applies to anything that alters which keychain is default, what is in the search list, or what material is installed — because those three are precisely the state the provisioning flow asserts, and two writers to one piece of state is the whole bug.

That is also why "but I know what I'm doing" is not an exception worth building. The problem was never competence. The problem is that one path asserts this state on a schedule and a human typing a command asserts it once, and the two disagree silently until something needs to sign.

How we proved it#

The incident is the proof, and the sequence is short enough to state exactly:

observed   a tmp keychain with a stale build id
action     removed it by hand
result     default keychain unset · no signing identity found, machine-wide
suspected  the provisioning repository (wrong — it had not changed)
offered    "Reset To Defaults" — a bigger change than the original mistake

What makes this worth writing down is not the command. It is that the diagnosis pointed at the wrong repository for most of a day, and the thing that would have saved it is not expertise — it is refusing to perform surgery on state that a flow already owns.

Also measured: the repair path works. The provisioning preflight now includes a keychain hygiene step that restores the default and the search list non-interactively, which means the situation that tempted the original command no longer requires anything from a person.

An unplanned confirmation. Writing this post tripped the guard. The draft quotes the forbidden command as evidence, and the pre-execution hook matched the string in the text — it cannot tell an article about a command from an attempt to run one. That is the cost of enforcing at the command layer, and it is the right trade: a guard that reads intent would be a guard that can be argued with.

What this generalizes to#

Machine-level state that a tool creates is state that tool should destroy. When you find yourself reaching for a command that mutates it by hand, the question is not whether the command is correct — it is why the tool's own flow is not doing it.

And when the system offers you a broad reset in the middle of an incident, decline. Any repair wider than the thing you broke is how a small mistake becomes a long day, and the dialog offering it has no idea what you were trying to fix.