What act can reproduce, and what it silently cannot

A local workflow runner executes your pipeline definition on your machine. For a containerised job this is close to the real thing. For a job that needs a specific operating system, it is not the real thing at all, and cannot be.

Both facts are fine. The problem is reporting them identically.

The case this post is about#

One workflow file with four jobs, run through a local runner on a macOS laptop:

$ python -m ci_local.classify .github/workflows/ci.yml
  PASS     build-android
  PASS     lint
  APPROX   test-unit  — cache action is a no-op locally
  NOT RUN  build-ios  — no macOS container
exit 1

Three of those four verdicts are honest and the fourth is the whole subject. build-ios declares runs-on: macos-latest, and there is no macOS container image — not on this machine, not on any machine. The local runner cannot execute that job and never will.

The question is what it prints when it cannot. A runner that reports three passes and stays silent about the fourth has told you your pipeline is green, which is a claim about four jobs made from evidence about three.

The symptom#

You adopt a local runner, iterate quickly, get everything green, push — and the macOS job fails on the real runner with something you never saw.

The failure is not the surprise. The surprise is that you believed you had tested it, which is a worse state than knowing you had not. Untested work gets attention at review; work you believe is tested gets waved through, and the belief came from your own tooling printing a success line.

The subtler version shows up weeks later. Someone adds a step to the macOS job, the local run says nothing is wrong because it never executed that job at all, and the gap between what the tool checked and what the reader assumed has quietly widened with nobody tracking it.

Why the obvious fix failed#

The obvious fix is to make the local runner handle everything. Add a macOS image, emulate the toolchain, stub what is missing.

This fails on physics rather than effort. Xcode, the Apple toolchains, code signing against the system keychain and the runner's own disk layout are not containerisable on a Linux host. You can approximate pieces, and an approximation of signing is precisely the thing that produces a green local run and a red remote one.

The second attempt is to skip the host jobs silently — run what can run, report success. That is the failure this article exists to name. A success line that covers four of six jobs, printed without saying so, is a measurement you cannot act on, because nothing distinguishes it from a success line covering all six.

The third is to refuse to run locally at all unless everything can run. Consistent, and it throws away the large real benefit of reproducing the containerised jobs because you cannot reproduce the rest.

The actual mechanism#

Run what you can, declare what you did not, and never let the two look alike.

Containerised jobs are reproduced, not approximated. Same image, same step definitions, executed locally. The result is evidence, and it should be reported as a pass.

Host jobs get a native approximation where one is possible. On a macOS workstation you can run the same build commands against the same local toolchain. That is genuinely useful — it catches compile errors, missing files and most logic mistakes — and it is not the runner's environment. It gets reported as approximate, never as a pass.

Jobs with no local path are skipped with a warning that names them. Windows on a Mac, a job needing a self-hosted runner, anything requiring hardware you do not have. The output must say which jobs were skipped and why, by name.

The summary carries coverage, not just a verdict. "4 reproduced, 1 approximate, 1 skipped (windows)" is actionable. "All checks passed" over the same run is not.

The middle socket is the one worth staring at. It is filled — something is there, it is the right shape, it occupies the position a result belongs in. It is also plainly not the same component as the one beside it, and that visible difference in substance is exactly what a summary line has to preserve when it collapses three states into text.

The fix#

The discipline that makes it work is a vocabulary with three values instead of two.

Most tooling has pass and fail. What this needs is pass, approximate and not-run — and the last two must be visually distinct from the first in whatever reads the output, including a human skimming at the end of a long day. The moment approximate renders as a green tick, the distinction is decorative.

This is the same shape as a judge that can return could not determine. A system with only two outcomes will classify everything into one of them, and the classification it chooses for "I did not check this" is almost always the optimistic one, because that is the one that lets the pipeline continue.

Here is the whole of it. The runner classifies each workflow job before anything runs, and the report prints what it could not do:

# ci_local/classify.py
APPROX = {"macos-latest": "no macOS container", "windows-latest": "no Windows container"}

def classify(job: dict) -> tuple[str, str]:
    runs_on = job.get("runs-on", "")
    if runs_on in APPROX:
        return "not-run", APPROX[runs_on]
    if job.get("uses"):                         # a reusable workflow: act cannot resolve it
        return "approximate", f"reusable workflow {job['uses']} inlined by hand"
    if any(s.get("uses", "").startswith("actions/cache") for s in job.get("steps", [])):
        return "approximate", "cache action is a no-op locally"
    return "pass", ""

def report(jobs: dict) -> int:
    rows = {name: classify(j) for name, j in jobs.items()}
    for name, (verdict, why) in sorted(rows.items()):
        mark = {"pass": "PASS", "approximate": "APPROX", "not-run": "NOT RUN"}[verdict]
        print(f"  {mark:<8} {name}" + (f"  — {why}" if why else ""))
    # NOT-RUN is not a pass. The exit code says so, or the distinction is decorative.
    return 1 if any(v == "not-run" for v, _ in rows.values()) else 0

On this repo's own workflow set:

$ python -m ci_local.classify .github/workflows/ci.yml
  PASS     build-android
  PASS     lint
  APPROX   test-unit  — cache action is a no-op locally
  NOT RUN  build-ios  — no macOS container
exit 1

build-ios is the line that matters. Four jobs ran green locally and one of them never executed — the exit code refuses to call that a pass, which is the entire difference between this and a checkmark.

Two details in that classifier are the whole of its usefulness. The reusable-workflow branch exists because uses: is the most common way a job quietly stops being reproducible — act cannot resolve another repository's workflow, so what runs locally is whatever somebody inlined by hand, and that is an approximation whether or not anybody wrote it down. The cache branch is there for a subtler reason: a cache action that no-ops locally does not fail, it simply makes every local run a cold-cache run, so a job that passes locally and fails in CI on a stale cache has been approximately reproduced in a way that hides exactly the class of failure caches cause.

What the three-value vocabulary buys is that the report can be read by somebody who was not there. PASS across the board on a repo with iOS jobs would be a lie that takes an afternoon to discover; NOT RUN build-ios — no macOS container is a sentence that costs nothing to read and cannot be misunderstood. The exit code carries the same information for anything downstream, which is what stops a green local run from being treated as a green CI run.

The cost of the vocabulary is that somebody has to maintain the APPROX table, and it is the kind of table that goes stale quietly — a runner label changes, a new action is adopted, and a job that should be flagged starts reporting PASS. I do not have a good answer for that beyond reading it when the workflow file changes, which is a dependency on attention of exactly the sort this post otherwise argues against.

What I would not give up is the exit code. A classifier that prints an honest report and still exits zero is a classifier whose honesty is optional: the pipeline continues, the next step treats the run as green, and the only thing standing between an unrun iOS job and a release is whether a person happened to read the middle of the output.

The corollary is that a skipped job is not a smaller pass. It is an absence of information, and the output should make a reader slightly uncomfortable about it. If the summary reads as clean when a third of the matrix never executed, the vocabulary has already failed.

There is a practical test for whether you have got this right, and it takes one minute. Delete a job from your local run — any job — and look at the output. If the summary still reads as success, with nothing visibly different, then your tooling is reporting a verdict about a set it never names, and the next person to trust it will trust it about a smaller set than they think.

The same test catches the version of this that creeps in later. Tooling tends to grow filters: run only the jobs that changed, only the fast ones, only the ones matching a pattern. Each filter is reasonable and each one silently shrinks the covered set, so a summary that does not name its scope becomes progressively less true without any single change being wrong.

How we proved it#

The measured part is what the local runner can and cannot touch, and it is a clean split.

linux / containerised    reproduced faithfully — same image, same steps
macos / xcode            NOT reproducible on a linux container runner
windows                  no local path on this machine
handling                 reproduce · approximate · skip-and-warn, never fake

Measured, and the reason the vocabulary matters: the same session that established this also produced a different confident-but-wrong green — an SEO audit reporting all 113 posts declare og:image while the content directory held 145 and the build it audited was two days old. The check ran, passed, and described the wrong artifact. Coverage and verdict are separate facts, and a tool that reports only the second will eventually report it about something you did not mean.

What I have not solved: the macOS gap itself. A native approximation on a developer machine catches a lot and does not catch signing, provisioning or anything depending on the runner's exact image. Those still cost a real cycle to verify. The honest position is that the local runner converted most failures from forty-five-minute round trips into two-minute ones and left a minority untouched — which is a large win reported accurately, rather than a total win reported dishonestly.

What this generalizes to#

Whenever you build a faster stand-in for a slow system, the stand-in will cover part of the surface. The engineering question is the easy half; the reporting question is where it goes wrong.

Say what you reproduced, what you approximated and what you never touched — by name, in the summary, every run. Give the output a third value so that "not checked" cannot be rendered as "fine". And hold the line that an approximation never prints as a pass, because the entire value of a fast local loop is the confidence it gives you, and confidence you have not earned is the one failure mode that makes the tool worse than not having it.