The onboarding doc that doubled as a test of our own setup

TL;DR#

Our setup doc was 40 steps long and every new engineer needed two to four days and a pair session to get running. The usual response is to improve the doc.

The doc was an accurate description of a bad setup. Forty manual steps, eleven of which existed only because something was undocumented, broken, or required a credential nobody could self-service. Making the prose clearer would have been polishing a symptom.

We turned the doc into a script, and the script into a CI job. The job runs on every pull request from a clean container, so the setup cannot rot — and the step count fell from 40 to 3, because a step that has to run in CI gets automated rather than explained.

The case this post is about#

CONTRIBUTING.md, 40 numbered steps. A representative stretch:

## Setup

12. Install JDK 17 (we use Temurin; 21 will not work — see note in step 31)
13. Set JAVA_HOME. On macOS with Homebrew this is usually
    /opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
14. Install the Android SDK. Accept the licences (this fails silently if you skip it)
15. Create local.properties with sdk.dir pointing at your SDK
16. Ask someone on the team for the staging API key and put it in local.properties
17. Run ./gradlew build — this will fail the first time, run it again
18. If you see "Could not resolve com.internal:*", you need the internal Maven
    credentials; ask in #eng-help
19. Install the pre-commit hooks: pre-commit install
20. The iOS build needs CocoaPods: sudo gem install cocoapods
    (if you are on Apple Silicon see the note at the bottom)

Steps 17 and 18 are the tell: "will fail the first time, run it again", and "ask in #eng-help".

The symptom#

Measured onboarding time for the last six joiners:

engineer   time to first green local build   pair sessions needed
A                   2.5 days                        2
B                   4 days                          3
C                   1.5 days                        1
D                   3 days                          2
E                   2 days                          2
F                   3.5 days                        3
median              2.75 days                       2

Where the time actually went, from asking each of them:

waiting for a credential (key, Maven token, repo access)   ~40%
a step that did not work as written                        ~25%
a step missing entirely (discovered by failure)            ~20%
actual installation and download time                      ~15%

Sixty percent of onboarding was blocked on access or on the doc being wrong — not on anything inherent to a large codebase.

And the doc was drifting faster than it was read. Between two joiners, three steps became obsolete and nobody updated them, because the person who changed the build had no reason to open CONTRIBUTING.md.

The compounding cost:

6 joiners × 2.75 days                       16.5 engineer-days
+ 13 pair sessions × ~1.5h × 2 people       ~5 engineer-days
= ~21 engineer-days per 6 hires

Why the obvious fix failed#

"Rewrite the doc more clearly." We did, twice. The second rewrite was genuinely better prose about the same 40 steps, and the median went from 2.75 days to 2.5. A clear description of a 40-step process is still a 40-step process.

"Add a troubleshooting section." It grew to fourteen entries. Each one documents a failure that could have been prevented — "if the SDK licences are not accepted you get this error" is a case for accepting them in a script, not for a FAQ entry.

"Pair with every new joiner for the first day." Effective and it is the cost, not the fix — two engineer-days per hire, and it hides the problem by making a person the workaround.

"Record a setup video." Goes stale faster than the doc and cannot be diffed. We made one; it was wrong within a month and nobody noticed because a video does not fail a build.

The actual mechanism#

A setup doc is an untested program with a human interpreter.

It has all the properties of code — ordered steps, conditionals ("if you are on Apple Silicon"), error handling ("run it again"), environment assumptions — and none of the guarantees:

Nothing executes it, so nothing detects rot. A step that stopped working stays in the document until a human hits it. Our three obsolete steps had been wrong for two months.

The human interpreter has to resolve ambiguity. "Usually /opt/homebrew/..." requires judgement from the person with the least context in the organisation.

And the failure mode is a question in a chat channel, which is invisible as a cost. Nobody tracks "hours spent in #eng-help", so the process looks cheaper than it is.

Which gives the reframing that fixed it:

The property that fixes it is making the document the thing CI runs, rather than a description of what CI runs. Each step is a fenced block with a check attached, and a runner executes them in order:

import re, subprocess

STEP = re.compile(
    r"^### (?P<title>.+?)\n+"
    r"```bash\n(?P<run>.*?)```\n+"
    r"```verify\n(?P<verify>.*?)```",
    re.M | re.S,
)

def run_doc(markdown: str, dry_run: bool = False) -> int:
    steps = list(STEP.finditer(markdown))
    if not steps:
        # A doc with no executable steps passes every check trivially. That is the
        # failure this whole exercise is about, so it is an error, not a pass.
        raise SystemExit("no executable steps found — this doc cannot rot-check itself")

    failed = 0
    for i, step in enumerate(steps, 1):
        title = step["title"]

        if not dry_run:
            subprocess.run(step["run"], shell=True, check=False)

        # The verify block is the part a prose doc has no equivalent for: it asserts the
        # state the step CLAIMS to produce, so a step that silently stopped working fails
        # here rather than in a new hire's afternoon.
        check = subprocess.run(step["verify"], shell=True, capture_output=True, text=True)
        if check.returncode == 0:
            print(f"  {i:>2}. ok       {title}")
        else:
            failed += 1
            print(f"  {i:>2}. FAILED   {title}")
            print(f"      {check.stderr.strip() or check.stdout.strip()}")

    print(f"\n{len(steps) - failed}/{len(steps)} steps verified")
    return 1 if failed else 0

Run against the existing document on the day it was converted, before anything was fixed:

$ python run_doc.py docs/ONBOARDING.md

   1. ok       Install the JDK
   2. ok       Clone the repository
   3. FAILED   Install the Android SDK via the standalone tools package
       sdkmanager: command not found — superseded by Android Studio's SDK manager
   4. ok       Create local.properties
   5. FAILED   Point ANDROID_HOME at /usr/local/share/android-sdk
       no such directory (Apple Silicon default is /opt/homebrew/share/android-sdk)
   6. FAILED   Run ./gradlew assembleDebug
       FAILURE: SDK location not found

9/12 steps verified

Three failures, in ninety seconds, naming the two months of rot that had been costing 2.75 days per hire. Note that step 6 fails because of step 5 — the runner does not stop at the first failure, because a new hire does not either, and the cascade is the thing worth seeing whole.

a doc that cannot be executed  →  rots silently, costs 2.75 days per hire
a script that CI runs          →  breaks loudly, on the PR that broke it

And a second-order effect that mattered more than the automation: a step that has to run in CI gets fixed rather than described. "Ask someone for the API key" cannot be a CI step, so it became a self-service credential. "Run it twice" cannot be a CI step, so the underlying race got fixed.

If the setup cannot be executed, it will be wrong and you will not know. Making it a script that CI runs converts documentation rot into a build failure — and forces the eleven steps that exist because something is broken to be fixed.

The fix#

One script, run by CI from a clean container, and a doc that is three lines long.

1. The script, idempotent and verbose about what it is doing:

#!/usr/bin/env bash
# scripts/setup.sh — the ONLY setup path. CI runs this on every PR.
set -euo pipefail

log() { printf '\n▸ %s\n' "$*"; }

log "Checking the toolchain"
./scripts/lib/require-jdk.sh 17          # installs via the toolchain resolver if absent
./scripts/lib/require-android-sdk.sh     # installs + accepts licences non-interactively

log "Materialising credentials"
# self-service: an OIDC exchange, not "ask someone in #eng-help"
./scripts/lib/fetch-dev-credentials.sh   # writes local.properties; no secret in any doc

log "Warming the build"
./gradlew --quiet help                   # resolves dependencies; fails loudly if the proxy is down

log "Installing hooks"
pre-commit install --install-hooks

log "Verifying"
./gradlew :app:assembleDebug :shared:jvmTest
log "Setup complete. Run './gradlew :app:installDebug' to deploy to a device."

2. The doc, which is now three steps:

## Getting started

1. Install Docker (or, for a native setup, just git + a shell).
2. `git clone` and run `./scripts/setup.sh`
3. Open the project in Android Studio.

The script is the source of truth. If it fails, that is a bug — file it rather than
working around it, because CI runs the same script on every PR.

That last sentence is the cultural half of the fix. "Working around it" is what kept the 40 steps alive.

3. CI runs it from a clean container, which is what stops the rot:

name: setup
on: [pull_request]
jobs:
  from-scratch:
    runs-on: ubuntu-latest
    container: ubuntu:24.04          # deliberately bare: no preinstalled JDK or SDK
    steps:
      - run: apt-get update && apt-get install -y git curl unzip   # the doc's prerequisites
      - uses: actions/checkout@v4
      - run: ./scripts/setup.sh      # the SAME script a human runs

A bare container is the point: using a prepared runner image would test the script against an environment no new joiner has.

4. And the eleven broken steps, fixed rather than documented:

step 16  "ask for the staging API key"      → OIDC-based self-service credential fetch
step 17  "run it twice"                      → a Gradle configuration race, fixed at source
step 18  "ask for Maven credentials"         → a read-only token issued by the same OIDC flow
step 14  "accept the licences"                → sdkmanager --licenses, scripted
step 20  "sudo gem install cocoapods"        → the Gradle plugin manages the pods
step 31  "JDK 21 will not work"              → a toolchain declaration, so the JDK is resolved
… 5 more

Six of the eleven were genuine defects that had been living as documentation. That is the finding I did not expect.

How we proved it#

Onboarding time for the next five joiners:

engineer   time to first green local build   pair sessions
G                   40 minutes                    0
H                   55 minutes                    0
I                   35 minutes                    0
J                   2 hours                       1   ← an unsupported Linux distro; script fixed
K                   30 minutes                    0
median              40 minutes                    0

Median 2.75 days to 40 minutes, and zero pair sessions in four of five. Engineer J's two hours produced a script fix, which is the loop working.

That CI catches the rot, which is the durable property:

setup.sh failures caught on PRs, 6 months:   9
  a new required credential not in the script:   3
  an SDK version bump that broke the install:    2
  a removed Gradle task the verify step used:    2
  a changed internal proxy URL:                  2

Nine setup breakages, each caught on the pull request that caused it rather than by the next hire. Previously every one of those would have been discovered by a person, days later, with no link back to the change.

And the steps, counted:

documented manual steps:   40 → 3
troubleshooting entries:   14 → 2
lines in CONTRIBUTING.md: 412 → 38

The cost of building it:

writing setup.sh and the lib scripts:    3 days
fixing the 6 underlying defects:         4 days
CI job:                                  2 hours
total:                                  ~7 days

saved, per 6 hires:                     ~21 engineer-days

Seven days to build, breaking even inside the first three hires — and the six defect fixes benefited everyone, not only new joiners.

One honest limit: the script works on Ubuntu and macOS, and engineer J's distro needed a fix. We have not tried to support every environment, and the doc now says which two are supported — which is a narrowing that is at least stated, rather than the previous situation where it worked on whatever the last person's machine was.

What this generalizes to#

Documentation that cannot be executed will be wrong, silently. A runbook, a setup guide, a deployment procedure, a disaster-recovery plan. If nothing runs it, nothing detects that it has rotted, and the first person to find out is in the worst position to fix it.

A long process is usually hiding defects, not complexity. Eleven of our forty steps existed because something was broken or inaccessible. Shortening the doc was impossible; fixing the underlying problems made the doc short as a side effect.

Automation forces the fix that documentation permits you to avoid. "Ask someone for the key" is a valid sentence and not a valid script line. The constraint of having to execute it is what turned six documented workarounds into six fixes.

Run it from a clean environment. A prepared CI image reproduces a machine nobody onboarding has. The bare container is the only version of this test that is testing the real thing.

Related#

• A stale doc does more damage than a missing one — confidently wrong documentation
• The architecture decision nobody wrote down, re-litigated four times — documentation worth keeping
• Seven modules, seven copies of the same Gradle block — configuration that drifts

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.