Forty-five minutes per CI guess is not a feedback loop

A check goes red. You read the log, form a theory, change one thing, push, and wait forty-five minutes to find out whether the theory was right.

It usually is not. That is fine — theories are cheap. What is not fine is that each one costs most of an hour, so a problem needing six iterations costs a day, and the day is almost entirely waiting.

The case this post is about#

One cycle, repeated until a day was gone:

loop            push → wait ~45m → one red check → adjust → repeat
iterations      enough to consume a day
actual fix      small
cost structure  waiting, not thinking

Each iteration was individually reasonable. A check went red, the message named one thing, that thing got fixed, and the next push surfaced the next one — because CI runs the steps in order and stops at the first failure.

The fix at the end of it was small. What cost the day was that every fact about the build cost forty-five minutes to obtain, and the facts arrived one at a time.

This post follows that loop, because the thing that ends it is not a faster pipeline.

The symptom#

The loop looks productive from the outside. Commits are landing, the pipeline is running, somebody is clearly working on it.

What is actually happening is that the feedback interval has grown longer than the thinking interval, which inverts how you work. When feedback is fast you test the cheapest hypothesis first, because being wrong costs nothing. When feedback costs forty-five minutes you start batching — changing four things at once to avoid another wait — and batching destroys the one property that made the loop diagnostic. A green build after four changes tells you almost nothing about which one mattered, and a red one tells you less.

The second symptom is worse and quieter: you stop reading the log carefully. After the fourth round trip, the temptation is to scan for the line that looks familiar and push the obvious fix, which is how a session ends with six commits and no understanding.

Why the obvious fix failed#

The obvious fix is to make CI faster. Cache more, parallelise, split the job.

It helps and it does not change the shape. Halving forty-five minutes still leaves a loop too slow to think in, and the work required is substantial. You can spend a week optimising a pipeline and still be unable to iterate on it.

The second attempt is to add debug output and push again — the instinct to instrument. This is the one that feels most like progress and is the most expensive, because the instrumentation itself costs a full cycle before it tells you anything, and usually needs two rounds to be useful.

The third is to reproduce it by hand: run the same commands locally and see. Closer to right, and it fails on fidelity. The reason the job failed on the runner and not on your machine is the difference between the two environments, so running the commands in your environment reproduces the thing that already worked.

The actual mechanism#

Reproduce the job, not the commands, and do it in two tracks at once.

Containerised runners can be reproduced faithfully. A Linux job is a container with a defined image and a known set of steps. A local runner executes that definition against the same image, so the thing you reproduce is the job rather than an approximation of it. The iteration cost drops from forty-five minutes to the length of the step you are debugging.

Host-dependent runners cannot. A macOS job needs macOS, Xcode, a specific toolchain and the runner's own disk layout. No container reproduces it. Pretending otherwise is the failure mode this whole approach exists to avoid, because a local pass that does not correspond to the real environment is worse than no information — it is confident misinformation.

So run both tracks in parallel. Containerised jobs go to the local runner for real fidelity. Host jobs get a native approximation on your own machine — the same build commands, the same toolchain where you have it — and are explicitly labelled as approximate.

Mechanical failures auto-heal; the rest you fix at source. A missing baseline regeneration, a formatting violation, a lockfile that needs updating — these have one correct fix and no judgement involved. Everything else is a real bug and gets read.

The engraving matters more than the channels. A local run that passes for a containerised job is evidence; a local run that passes for a host job is a guess wearing the same green tick, and the moment those two look alike in your output you have built a machine for producing false confidence.

The fix#

The part that makes this stick is not the tooling. It is the refusal to push until local is green.

Without that, the local runner becomes one more thing you try before pushing anyway, and the loop survives. With it, the loop is mechanically impossible: there is a marker that says a CI fix is in flight, and the push path is blocked while it exists and local has not gone green. The marker clears

the instant local passes, so the block is self-limiting rather than something anybody has to remember to lift:

marker="$STATE/cifix-$SLUG.json"

# Called by the push path, BEFORE anything leaves the machine.
assert_local_green() {
  [ -f "$marker" ] || return 0                       # no CI fix in flight — nothing to say

  # A marker older than the session is stale, not a block. Guards that outlive
  # their reason are guards people disable.
  age=$(( $(epoch) - $(json_get "$marker" started_at) ))
  if [ "$age" -gt 14400 ]; then rm -f "$marker"; return 0; fi

  if [ "$(json_get "$marker" local_green)" = "true" ]; then return 0; fi

  cat >&2 <<MSG
refusing to push: a CI fix is in flight and local is not green.
  reproduce locally:  ci-pr-fix --pr $(json_get "$marker" pr)
  the loop this prevents: push -> wait 45 min -> one more fix -> push
  stuck marker:       ci-pr-fix --clear
MSG
  return 2
}

The block, and the moment it lifts:

$ git push
refusing to push: a CI fix is in flight and local is not green.
  reproduce locally:  ci-pr-fix --pr 412
  the loop this prevents: push -> wait 45 min -> one more fix -> push
  stuck marker:       ci-pr-fix --clear

$ ci-pr-fix --pr 412
  act: build-android PASS · lint PASS · test-unit PASS
  local green — marker cleared

$ git push                      # proceeds

The staleness escape matters as much as the block. Without it, one abandoned fix session leaves a marker that blocks every push until somebody finds the file — and that is how a guard earns a reputation that makes the next one unwelcome.

The deeper reason the loop persists without a mechanical block is that each individual push is rational. You have made a change you believe fixes the failure; the only authoritative test is CI; pushing is how you run it. Every step of that reasoning is sound, and the aggregate is a day spent waiting forty-five minutes at a time to learn things a local run could have told you in four.

What breaks it is not discipline but availability: the loop survives exactly as long as pushing is the easiest way to find out. Once there is a local reproduction that runs the same jobs in a Linux container, pushing stops being the fastest feedback and the behaviour changes on its own — the guard is there for the cases where the local path is awkward and the temptation returns.

The honest limit is that local reproduction is not complete. Container-based runners cannot run macOS or Windows jobs, so those genuinely do require a push to verify. The guard knows this: it blocks on the jobs that can be reproduced and reports the others as unverified rather than pretending they passed. A guard that claimed full local coverage would be the same lie in a new place.

There is one more property worth copying: the marker records which pull request the fix is for. That sounds like bookkeeping and it is what makes the block comprehensible three hours later, when whoever hits it has forgotten there was a fix in flight at all. A refusal that says "a CI fix is in flight" prompts a search; one that names the PR prompts a decision.

The same applies to the clear command being in the message. A guard that can be stuck and does not tell you how to unstick it gets disabled wholesale the first time it misfires, and the disabling is permanent in a way the misfire was not. itself the moment local passes.

That sounds heavy-handed and it is the opposite of a rule you have to remember at the exact moment you are least inclined to. The temptation to push-and-see is strongest after the third failure, and a guard that refuses then is worth more than any amount of discipline.

Two things it must never do. It must not let you fake a macOS check green — those skip with a warning and are confirmed on a real runner. And it must have a human override that lives outside the agent's reach, because a genuinely stuck marker should cost one deliberate command rather than an hour of fighting your own tooling.

How we proved it#

The motivating case is a day, and the shape of it is the evidence.

A red pipeline was being fixed by the loop in the title — push, wait, read one check, adjust, push. Each iteration cost roughly forty-five minutes, most of it idle. The eventual fix was small. The cost was almost entirely the waiting, multiplied by the number of wrong theories, which was not an unreasonable number of theories.

loop            push → wait ~45m → one red check → adjust → repeat
iterations      enough to consume a day
actual fix      small
cost structure  waiting, not thinking

What replaced it: reproduce the failing checks locally and in parallel — the containerised ones faithfully, the host ones approximately and labelled as such — auto-heal the mechanical class, fix the rest at source, and loop until local is green. Then push once.

The guard is the part I would not skip. Local reproduction without it is a tool you use when you remember. With it, the push-and-pray loop cannot happen, because the push is blocked while a fix is in flight and unproven.

What I have not solved: Windows and macOS fidelity. Both are skipped locally with a warning and confirmed on the real runner, which means a class of failures still costs a real cycle. That is a genuine hole, not a detail — it is just a much smaller hole than the one where every failure cost a cycle.

What this generalizes to#

When a feedback loop is slower than the thinking it is supposed to support, the fix is not to make the loop faster. It is to move the feedback somewhere else entirely.

Find which parts of the slow system can be reproduced faithfully and reproduce those; be explicit that the rest are approximations and never report one as a pass. Then block the slow path while the fast one is unproven, because the moment you most want to skip the fast path is the moment it is most valuable — and a rule you have to remember will not survive that moment.