Two agents, one working tree, one disaster

Ten agents, one repository, each told to fix a different feature module. Nine of them finish. The tenth reports success and its change is not in the diff.

Nothing failed. They were all writing to the same working tree.

The case this post is about#

A fan-out over ten feature modules in one Kotlin Multiplatform project, each agent asked to add a missing handler to its own module:

feature/profile      agent 1
feature/settings     agent 2
feature/progress     agent 3
...
feature/onboarding   agent 10

Ten disjoint directories. No two agents touch the same file, which is the property that makes this look safe and is not the property that matters.

The thing they share is not a file. It is HEAD, the index, and settings.gradle.kts — and every one of those ten agents needs to add an include(":feature:<name>") line to the last of those.

The symptom#

The first shape is the lost write, and it is the one that produced this post:

$ git status --short
 M feature/profile/src/commonMain/kotlin/ProfileHandlers.kt
 M feature/settings/src/commonMain/kotlin/SettingsHandlers.kt
 M settings.local.gradle.kts

Three modified files from a ten-agent run. Seven agents wrote files that are not there. Each one reported completion, each one had genuinely written its file, and each write was clobbered by the next agent reading settings.local.gradle.kts, appending its own line to the version it had read, and writing the whole file back.

The second shape is the one that cost real time: an agent running a build mid-run and getting somebody else's compile error. Agent 3 runs ./gradlew :feature:progress:compileKotlin, the compile fails on a syntax error in feature/onboarding, and agent 3 sets about diagnosing a file it has never opened. It is a correct diagnosis of a real error belonging to nobody.

The third is the one that looks like a tooling bug. Two agents stage changes, and whichever commits first commits both. The second agent's commit is empty, and git commit on an empty index does not fail loudly in a way an agent reads as a failure — it reports nothing to commit, which is exactly what a successful second commit of already-staged work would look like.

Why the obvious fix failed#

Partition the files. The instinct, and it is already true — the ten agents had disjoint file sets by construction. It fixes nothing because the shared state is not the files. HEAD is shared, the index is shared, and one file nobody thought about (settings.local.gradle.kts) is written by all ten.

Lock the shared file. Correct, and it only works for the file you thought of. I added a lock around the Gradle settings file and the next run lost writes to gradle/libs.versions.toml, which three agents needed to add a dependency to. The list of files ten agents might both touch is not a list anyone finishes.

Serialise the writes and keep the reads parallel. Appealing, and it does not survive the build step. An agent that compiles to check its own work is reading the whole tree, so a parallel read during somebody else's half-finished write produces the phantom compile error above. The read is the problem, not just the write.

Run them serially. This works. It is also the thing the fan-out existed to avoid, and for ten modules at roughly four minutes each it is the difference between seven minutes and forty.

The actual mechanism#

Give each agent its own working tree, pointed at the same object database.

That is what git worktree is: a second checkout of the same repository, with its own HEAD, its own index, and its own files on disk, sharing the objects and refs. Two agents in two worktrees cannot see each other's uncommitted state at all — not the files, not the index, not a half-written Gradle settings file.

The non-obvious part is when it is worth it, because isolation is not free and the cost is usually quoted wrong.

The orchestration contract I was working against advertises worktree isolation as "expensive (~200-500ms setup + disk per agent)". That figure is what I planned around, and planning around it is why a twelve-agent run took four minutes longer than I had budgeted for before any agent did anything.

The second non-obvious part is the cleanup, and it cuts the other way. A worktree whose agent changed nothing is pure waste — setup cost, disk, and teardown, for an agent that read some files and reported a finding. Most read-only fan-outs are exactly that shape.

The fix#

The decision is a function of what the agents write, not of how many there are:

type AgentPlan = {
  id: string;
  writes: string[];          // paths this agent will mutate
  runsBuild: boolean;        // does it read the whole tree?
};

type Isolation = { mode: "shared" | "worktree"; why: string };

// Measured on this repo, not the documented figure. See "How we proved it".
const WORKTREE_SETUP_MS = 4_300;
const WORKTREE_DISK_MB = 364;

function decide(plan: AgentPlan[], diskBudgetMb: number): Isolation {
  const writers = plan.filter((p) => p.writes.length > 0);

  // A read-only fan-out shares. Nothing to corrupt, and the setup is pure waste.
  if (writers.length === 0) {
    return { mode: "shared", why: `${plan.length} agents, none of which write` };
  }
  if (writers.length === 1) {
    return { mode: "shared", why: "exactly one writer — nothing to interleave with" };
  }

  // The overlap test is NOT "do two agents write the same file". It is whether any
  // writer's output is visible to another agent. A build reads the WHOLE tree, so an
  // agent that builds overlaps with every writer by definition.
  const builders = plan.filter((p) => p.runsBuild);
  if (builders.length > 0 && writers.length > 1) {
    return {
      mode: "worktree",
      why: `${builders.length} agent(s) build while ${writers.length} write — a build reads every path`,
    };
  }

  const seen = new Map<string, string>();
  for (const p of writers) {
    for (const path of p.writes) {
      const prior = seen.get(path);
      if (prior) return { mode: "worktree", why: `${prior} and ${p.id} both write ${path}` };
      seen.set(path, p.id);
    }
  }

  // Disjoint writes, no builds. The one case where parallel-in-place is safe —
  // and it is rarer than it looks, because shared manifests are easy to forget.
  const cost = writers.length * WORKTREE_DISK_MB;
  return cost > diskBudgetMb
    ? { mode: "shared", why: `disjoint writes; ${cost} MB of worktrees exceeds the ${diskBudgetMb} MB budget` }
    : { mode: "shared", why: "disjoint writes, no builder — in-place is safe" };
}

And the budget check that decides whether the fan-out is worth running at all:

function worthFanningOut(plan: AgentPlan[], perAgentSeconds: number, cap: number) {
  const iso = decide(plan, 8_000);
  const waves = Math.ceil(plan.length / cap);

  const serialSeconds = plan.length * perAgentSeconds;
  const setupSeconds = iso.mode === "worktree"
    ? (plan.length * WORKTREE_SETUP_MS) / 1_000      // serialised: git takes a lock
    : 0;
  const parallelSeconds = setupSeconds + waves * perAgentSeconds;

  return {
    isolation: iso.mode, why: iso.why,
    setupSeconds: Math.round(setupSeconds),
    serialSeconds, parallelSeconds: Math.round(parallelSeconds),
    verdict: parallelSeconds < serialSeconds
      ? `fan out — saves ${Math.round(serialSeconds - parallelSeconds)}s`
      : `run serially — isolation costs ${Math.round(parallelSeconds - serialSeconds)}s more than it saves`,
  };
}

The ten-module case, and the read-only sweep it is easy to confuse with:

const modules = ["profile", "settings", "progress", "onboarding", "home", "ledger",
                 "search", "export", "sync", "billing"];

console.log(worthFanningOut(modules.map((m) => ({
  id: m,
  writes: [`feature/${m}/src/commonMain/kotlin/${m}Handlers.kt`, "settings.local.gradle.kts"],
  runsBuild: true,
})), perAgentSeconds = 240, cap = 12));

console.log(worthFanningOut(modules.map((m) => ({
  id: m, writes: [], runsBuild: false,
})), perAgentSeconds = 40, cap = 12));

{
  isolation: "worktree",
  why: "10 agent(s) build while 10 write — a build reads every path",
  setupSeconds: 43,
  serialSeconds: 2400,
  parallelSeconds: 283,
  verdict: "fan out — saves 2117s"
}
{
  isolation: "shared",
  why: "10 agents, none of which write",
  setupSeconds: 0,
  serialSeconds: 400,
  parallelSeconds: 40,
  verdict: "fan out — saves 360s"
}

Forty-three seconds of setup to save thirty-five minutes is an easy trade, and the arithmetic is worth having in front of you rather than inferred, because the same function says no to a case that feels identical:

console.log(worthFanningOut(
  ["profile", "settings"].map((m) => ({
    id: m, writes: [`feature/${m}/Doc.md`], runsBuild: false })),
  perAgentSeconds = 6, cap = 12));

{
  isolation: "shared",
  why: "disjoint writes, no builder — in-place is safe",
  setupSeconds: 0,
  serialSeconds: 12,
  parallelSeconds: 6,
  verdict: "fan out — saves 6s"
}

Two agents writing two documentation files need no isolation at all. Had the decision been "more than one writer means worktrees", those two agents would have paid 8.6 seconds of setup to save six.

How we proved it#

The setup cost is ten times the documented figure, on this repository. Three consecutive git worktree add --detach HEAD operations, timed:

worktree add #1: 5532 ms
worktree add #2: 4426 ms
worktree add #3: 4308 ms

And each one is 364 MB on disk:

364M    /var/folders/.../tmp.L12cTWc7re/wt1

The contract I was planning against says ~200–500ms. The measured figure is 4.3–5.5 seconds, which is roughly an order of magnitude out, and the reason is entirely about this repository rather than about git worktree — it is large, it has submodules, and a checkout has real files to write. A figure quoted in a tool's documentation is a figure measured on somebody else's repository.

That matters for the decision rather than just for the estimate. At 200ms, twelve worktrees cost 2.4 seconds and you would never think about it. At 4.3 seconds they cost 52, which is long enough that a fan-out of short tasks is slower than running them in sequence — and the crossover point sits in a range that short agent tasks actually occupy.

The teardown is conditional, and that part held. Three worktrees created, none modified, all removed, and git worktree list back to one entry. An unchanged worktree is reclaimed, so the 364 MB is transient for a read-only agent — which is an argument for the isolation being cheap in disk and does nothing about the 4.3 seconds, because the setup is paid before anybody knows whether the agent will write.

What I have not measured is the corruption rate. I know the ten-agent in-place run lost seven writes, once. I do not have a count across runs, because after the first one I stopped doing it — which means the benefit side of this is a single observation and a mechanism, not a rate. The mechanism is solid enough that I am not going to gather the sample.

And a real limit: git worktree add takes a repository-level lock, so the setup does not parallelise. Twelve worktrees is twelve sequential 4.3-second operations, and that is why setupSeconds above multiplies rather than taking a maximum. For a fan-out wide enough to matter, the isolation setup is itself a serial phase at the front of your parallel one.

What this generalizes to#

The test for whether parallel workers need isolation is not do they write the same file — it is can one worker observe another's unfinished state. Those are very different questions, and the first one is the one that gets asked.

A build reads every path, so one building worker overlaps with every writing worker. A shared manifest, lockfile or settings file is written by everyone and appears in nobody's plan. A worker that reads the tree to check its own work has made itself a reader of everything. In each case the file-level partition is genuinely correct and the workers still interfere, because the unit of interference is the working tree and not the file.

So partition by observable state and then price the isolation on your own hardware, because the figure in the documentation was measured on a repository that is not yours — and the decision of whether to fan out at all can turn on exactly that number.