Engineering

Killing the Gradle daemon costs you thirty minutes

Killing a build daemon discards the configuration cache, build cache, file watching and JIT state at once. Forty seconds becomes half an hour.

MobileByteSensei 10 min read
A furnace at working heat, and the hours between cold and workable
gradlebuildtooling

An incremental build here takes about forty seconds. The same build, after the daemon has been killed, takes between fifteen and thirty minutes.

Nothing about the code changed. What changed is that four kinds of accumulated state were thrown away at once, and all of them have to be rebuilt before the first useful work happens.

The case this post is about

One build that was behaving strangely, and the command everybody reaches for:

refuse       stopping the daemon, killing the process, killing all JVMs
allow        status inspection; stopping an app on a device
configure    long idle timeout, caches, parallel, file watching — user-level
repair       config-cache incompatibilities at source, upstream when shared
escape       a human, in a real shell, deliberately

An incremental build here takes about forty seconds. The same build after the daemon has been stopped takes fifteen to thirty minutes, and the code is identical — four kinds of accumulated state were discarded at once.

The reflex works often enough to become a habit, and it costs half an hour every time it fires without cause. This post follows those five lines, because the second one is the detail that makes the refusal survivable: stopping an app on a device looks like the same command and is a different thing entirely.

The symptom

The visible version is simply that the build is slow, with no explanation available anywhere in the output. A cold build does not announce itself as cold. It prints the same task names in the same order as a warm one, just much further apart, so the experience is of a toolchain that has inexplicably become unusable.

The second thing is the reflex that caused it, and this is the part worth looking at honestly. A build misbehaves — a stale artifact, a confusing failure, something that smells like corrupted state — and the move everyone reaches for is to restart the daemon. It frequently appears to work. So the lesson learned is killing the daemon fixes things, which is true often enough to become a habit and expensive every single time.

A glory hole gone cold, which can be relit and cannot be hurried

The third is that you do not lose one thing, you lose four, and they are rarely counted together. The configuration cache, which stores the result of evaluating your build scripts. The build cache and incremental compilation state. File-system watching, which knows what has changed since last time. And the JVM's own warmth — a long-running daemon has JIT-compiled the hot paths of the build itself. Each of those is individually significant and they are discarded as a set.

The fourth is what it does to the diagnosis. Restarting makes the symptom go away without producing any information about it, so whatever was actually wrong is undiagnosed and will return. You have paid half an hour to postpone the question, and because the payment is in waiting rather than in effort, it does not feel like a cost at all.

Why the obvious fix failed

Just do not kill it. A rule enforced by memory, and the moment it needs to hold is the moment someone is frustrated with a build that is behaving strangely. Discipline is weakest exactly where this rule is load-bearing.

Make the daemon more reliable. Reasonable and mostly not yours to do. The daemon is generally fine; the thing that goes wrong is usually in the build scripts or in a plugin, which means "reliability" here is a diagnosis you are avoiding by restarting.

Accept the cold build. Defensible if it happened rarely. Multiply thirty minutes by the number of times a day somebody reaches for a restart and it becomes the single largest line item in the toolchain, which is a strange thing to pay for a mechanism that exists to make builds fast.

Kill it only when genuinely wedged. The rule everybody actually believes they are following, and it requires distinguishing wedged from slow or confusing — which is precisely the diagnosis you do not have at the moment you want to restart. The condition is unobservable at decision time, so the rule degrades to the first one.

The actual mechanism

Make the kill mechanically unavailable to anything automated, and fix the underlying causes at their source.

Those two halves are both necessary and the second is the one that makes the first tolerable. A block with no path to resolution just relocates the frustration.

Block the kill. The specific commands — stopping the daemon, killing the process, killing every JVM on the machine — are intercepted and refused. Not warned about: refused, because a warning in front of a reflex is not a control. Inspecting the daemon's status stays available, as does killing the application on a device, which is a different thing that happens to use a similar-looking command.

The furnace's door interlock, which is not a notice but a mechanism

Fix the thing that made you want to. The most common genuine cause is a configuration cache incompatibility — typically a task that captures the project object and uses it at execution time. The tempting response is to disable the configuration cache, and that trade is worse than it looks: the incompatibility is a real correctness problem in the task, and the cache was the only thing telling you about it. Disabling it buys a slower build and loses the signal.

Heat held in the mass, which is what the hours bought

Keep the warmth machine-global rather than per-project. A long daemon idle timeout, caching, parallel execution, file-system watching — these belong in the user-level configuration, so every project inherits them and no project can accidentally opt out. Warmth that has to be configured per repository is warmth that is missing in the repository you cloned this morning.

The sub-case that matters most: when the task causing the incompatibility is not yours to change — part of a shared template, or a third-party plugin — the fix still belongs at the source rather than in a local workaround. Patching it locally leaves everybody else with the defect and leaves you with a change that the next sync removes.

And there has to be an escape, because occasionally a daemon genuinely is stuck. It belongs to a human, explicitly, in a real shell — not to any automated actor, because an automated actor cannot tell wedged from slow either and will reach for it far more readily than a person would.

The fix

refuse       stopping the daemon, killing the process, killing all JVMs
allow        status inspection; stopping an app on a device
configure    long idle timeout, caches, parallel, file watching — user-level
repair       config-cache incompatibilities at source, upstream when shared
escape       a human, in a real shell, deliberately

The guard is a verb list, and the interesting part is what it deliberately leaves alone:

cmd="$1"

# ALLOWED, explicitly: inspecting the daemon, and stopping an APP on a device.
# Stopping an app under test has nothing to do with the build daemon, and blocking
# it would push people toward killing the daemon instead.
case "$cmd" in
  *"--status"*|*"force-stop"*) exit 0 ;;
esac

if printf '%s' "$cmd" | grep -qE 'gradlew +--stop|daemon-kill|killall +java'; then
  cat >&2 <<'MSG'
refusing: that stops the build daemon.
  A warm daemon + configuration cache + build cache is what makes incremental
  builds seconds instead of a 15-30 min cold reconfigure. Stopping it does not
  fix anything; it pays that cost again.
    wedged task?        fix the task - usually a Project capture at execution time
    config-cache error? repair at source; upstream it if the file is shared
    genuinely stuck?    a HUMAN, in a real shell, with the bypass exported
MSG
  exit 2
fi
exit 0

The pair that looks the same to somebody in a hurry and is not:

$ adb shell am force-stop com.example.app
    (allowed - kills the APP, which on-device verification requires)

$ ./gradlew --stop
refusing: that stops the build daemon.
  A warm daemon + configuration cache + build cache is what makes incremental
  builds seconds instead of a 15-30 min cold reconfigure...

Allowing the app-stop is not an oversight. A guard that blocks it makes on-device verification impossible, and somebody working around that restriction will reach for the daemon.

The thing to protect is the asymmetry: the block costs a few seconds of annoyance in the rare case where a restart was genuinely warranted, and it saves half an hour every time the reflex fires without cause. That ratio is what justifies a refusal rather than a prompt.

The ordering of the two halves matters in practice too. Shipping the block before fixing the underlying causes produces exactly the outcome you would expect: somebody hits a genuine configuration cache failure, cannot reach for the restart, and now has a problem with no path through it. The repairs have to land first, so that by the time the block exists the common reasons for wanting a restart have already stopped happening.

That is also the honest test of whether a guard like this is ready. If you cannot name what somebody should do instead in the three most common cases, the guard is premature — it is enforcing a rule whose alternative you have not built yet.

It is also worth stating what the block is not for. It is not protecting the daemon from the user; it is protecting the user from a habit formed by a dozen occasions where restarting appeared to help. The habit is the thing being interrupted, and habits do not respond to warnings.

How we proved it

The measurement is the argument and it is large enough that precision does not matter much. Incremental build, warm: around forty seconds. Full cold reconfigure and recompile after a daemon kill: fifteen to thirty minutes. That is not a tuning difference, it is a change in what kind of activity building is — forty seconds is something you do while thinking about the change, and half an hour is something you go and do something else during.

The configuration cache incompatibility is the demonstrable cause. A task that captures the project object fails the configuration cache explicitly, by name. Fixing the task restores the cache; disabling the cache hides the failure and keeps the slow build. Both paths are available and only one of them is a fix, which is what makes this a good example of the general pattern — the fast path and the diagnostic path are the same path.

The four kinds of state are separately verifiable, which is worth saying because it is what turns "the build is slow" from a feeling into something you can check. Configuration cache reuse is reported explicitly on each run. Incremental compilation shows in which tasks execute versus which are up to date. The daemon's own uptime is a query. None of this requires instrumentation you have to build — it is already printed or already available, and the only reason a cold build feels mysterious is that nobody looks at those three things together.

What I have not measured is how often the block actually fires. I know the cost of a cold start and I do not have a count of prevented kills, so the benefit side of the ratio is reasoned rather than observed. That is the same gap I keep finding in guards I have built: they are easy to justify and hard to evaluate, because a guard that works produces no events.

What I have not solved: the genuinely wedged daemon. The escape exists and it is a human decision, which means on the day it is needed the person has to recognise the condition, remember the override and type it. I have not made the wedged state detectable — there is no check that says "this daemon is not responding, a restart is warranted" — so the hardest case is still handled by judgement at the worst moment.

Four kinds of warmth, discarded together and rebuilt one at a time

And a boundary on the whole argument: these numbers are from one machine with one project. A smaller build has less to lose and a bigger one has more, and the ratio that makes a hard refusal obviously correct here would be a different conversation on a project where cold is ninety seconds. The principle holds; the severity is mine, not universal.

What this generalizes to

Anywhere a tool maintains expensive warm state, "restart it" is a diagnosis-avoidance move with a price tag, and the price is usually unmeasured because it is paid in waiting.

Dev servers, language servers, container layer caches, connection pools, test runners holding a loaded fixture set — all of them accumulate state that makes the second operation much cheaper than the first, and all of them have a restart that discards it. The restart works often enough to become reflexive, and each time it works it removes the evidence that would have explained the problem.

So when you reach for a restart, the useful question is what am I throwing away, and what was I about to learn? If the answer to the first is "several minutes of accumulated work" and to the second is "nothing, because I am not looking", that is the moment the habit is costing you — and it is worth making it hard to do, because by the time you are reaching for it you have already stopped diagnosing.

One post a week, no digest

New writing when it lands. No roundups, no course pitch, no "10 tips".

Follow by RSS