The config change that was really a deploy, and had none of the safeguards

TL;DR#

Code reaching production passes review, CI, a canary and a progressive rollout, with an automatic revert. Config reaching production passes a text box and a Save button.

Both change behaviour. One of them has every safeguard your organisation has built, and the other has none — which is why the worst outages in a mature system are so often "just a config change".

Ours was a connection-pool size, typed into a web form, applied instantly to every instance. The fix is to make config travel the same path as code: reviewed, validated, canaried, and revertible.

The case this post is about#

A runtime configuration service, used the way they are: a UI, a key, a value, immediate effect.

key:    db.pool.maxSize
old:    20
new:    200
reason: "pool exhaustion during peak, raising headroom"

The reasoning was sound — we were exhausting the pool. The change was made at 16:40 on a Tuesday by someone who had diagnosed the problem correctly.

The symptom#

16:40:02  config applied: db.pool.maxSize 20 → 200
16:40:03  all 48 instances begin opening connections
16:40:11  postgres: FATAL: sorry, too many clients already
16:40:11  every endpoint failing
16:41:30  on-call paged
16:52:00  identified as the config change
16:52:40  reverted to 20
16:54:10  recovered

Fourteen minutes of total outage. The database's max_connections was 500; 48 instances × 200 is 9,600.

Four things about that timeline:

It applied to all 48 instances at once. No canary, because the config service pushes to every subscriber. A code change making the same edit would have gone to 1 instance first.

Nothing validated the value. 200 was accepted as readily as 20. A static check against max_connections / instance_count would have rejected it in milliseconds.

There was no review. One person's correct diagnosis and incorrect arithmetic, with nobody in between. The same change in application.yaml would have had a reviewer.

And rollback was manual. Finding the cause took twelve of the fourteen minutes, because the config change was not in any deployment timeline — the incident dashboard showed no deploys, which actively misled the responders.

Why the obvious fix failed#

"Train people to be careful with config." The person was careful. They diagnosed accurately and made an arithmetic error about a global limit, which is exactly the error a validation rule exists to catch.

"Require a second person to approve in the UI." We added it, and it is better. An approver looking at 20 → 200 in a dialog has no more access to max_connections than the author did — review without the relevant context is a formality.

"Put the dangerous keys behind a runbook." Documentation adjacent to a text box. The text box still works.

"Move everything back into application.yaml." Correct for this key, and it throws away the reason runtime config exists: changing a timeout during an incident without a 20-minute deploy. The answer is not to abolish fast config, it is to make fast config safe.

The actual mechanism#

Two paths to production, with wildly different safety properties.

The code path has accumulated safeguards over years:

edit → review → CI (build, test, lint) → canary 1 instance → 10% → 50% → 100%
     → automatic revert on error-rate regression → full audit trail in git

The config path has one step:

edit → applied to 100% of instances, immediately

The gap is not an oversight — it is the point of a config system. Every safeguard the code path has costs latency, and config exists precisely so an operator can change behaviour faster than a deploy. The feature and the hazard are the same property.

Which means "add review and a canary to config" has to be applied selectively, or it removes the reason config exists. The separating question is what the value can reach:

rate limits, timeouts, log levels       bounded blast radius — keep the fast path
feature flags (percentage rollouts)     already gradual by construction — keep it
a flag that disables a payment provider
a value used as a divisor or an array bound
anything consumed at startup only       ← these are deploys wearing a config costume

The last row is the one that caught us, and it is worth stating separately: a value read once at boot does not take effect until something restarts, so the change appears inert, gets forgotten, and then lands during an unrelated deploy hours later — attributing the incident to whatever code shipped that afternoon.

And the critical asymmetry: config changes behaviour exactly as much as code does. A pool size, a timeout, a feature flag, a rate limit, a retry count, a routing weight — each one alters runtime behaviour as surely as an if statement, and often more globally.

Three specific gaps caused the fourteen minutes:

No validation. The value's valid range is a function of another system's limit, which nothing checked.

No progressive rollout. The blast radius was 100% from the first second, so there was no window in which a canary's error rate could have stopped it.

No visibility in the deployment timeline. The responders' first question — "what changed?" — was answered "nothing", because config lives in a different system from deploys.

Config is a deploy. If it changes behaviour in production, it needs the controls you built for changes that alter behaviour in production — the delivery mechanism is not what decides the risk.

The fix#

Config in git, validated in CI, rolled out progressively, and visible on the same timeline as deploys.

1. Config as a reviewed artifact, with a typed schema:

# config/production.yaml — reviewed, versioned, diffable
db:
  pool:
    maxSize: 20        # see validation: maxSize × instances must be < 60% of pg max_connections

// config/schema.kt — validation that runs in CI, with access to the real limits
object ConfigSchema {
    fun validate(c: Config, env: Environment) {
        val headroom = env.postgresMaxConnections * 0.6
        val demanded = c.db.pool.maxSize * env.instanceCount
        require(demanded <= headroom) {
            "db.pool.maxSize=${c.db.pool.maxSize} × ${env.instanceCount} instances = $demanded " +
            "exceeds 60% of postgres max_connections (${env.postgresMaxConnections})"
        }
        require(c.http.timeoutMs in 100..30_000) { "http.timeoutMs out of range" }
        // …one rule per key whose valid range depends on something else
    }
}

That single require would have rejected the change in CI:

❌ db.pool.maxSize=200 × 48 instances = 9600 exceeds 60% of postgres max_connections (500)

2. Progressive rollout, the same as code:

# the config service subscribes to the git ref and stages by cohort
rollout:
  stages: [{ instances: 1, hold: 2m }, { percent: 10, hold: 5m }, { percent: 100 }]
  abort_on:
    - metric: error_rate
      threshold: 1.5x_baseline

One instance for two minutes would have surfaced the too many clients error with 1/48th of the impact.

3. Config changes on the deployment timeline:

// the config service emits the same event type as a deploy
events.emit(ChangeEvent(
    kind = ChangeKind.CONFIG,
    ref = commitSha,
    summary = "db.pool.maxSize 20 → 200",
    author = author,
))

So "what changed?" has one answer in one place. This is the change that would have cut twelve minutes off the incident.

4. And an explicit break-glass path, because the reason runtime config exists is real:

# emergency: bypasses review, NOT validation. Audited, time-limited, auto-reverting.
$ configctl emergency-set db.pool.maxSize 40 \
    --reason "pool exhaustion, incident-4812" --ttl 2h
  ✓ schema validation passed (40 × 48 = 1920 ≤ 300 … ✗)
  ❌ rejected: exceeds 60% of max_connections

Note what break-glass skips and what it does not: review and rollout, never validation. An emergency is a reason to move fast, not a reason to permit an arithmetically impossible value.

How we proved it#

That the validation catches the original change:

$ configctl validate --env production config/production.yaml   # with maxSize: 200
❌ db.pool.maxSize=200 × 48 instances = 9600 exceeds 60% of postgres max_connections (500)
exit 1

That the canary limits the blast radius, tested by deliberately staging a bad-but-valid value:

db.pool.maxSize 20 → 90   (90 × 48 = 4320 > headroom, but passes a naive range check)
  stage 1 (1 instance):   error_rate 11.4× baseline → ABORTED after 38s
  instances affected:     1 of 48
  user-visible errors:    0.4% for 38s

Compare the original: 100% of instances, 14 minutes, total outage.

And the audit, which found the scale of the gap:

runtime config keys:                            214
keys whose value can cause an outage:             37
keys with a validation rule (before):              0
keys with a validation rule (after):              37
config changes in the previous 12 months:        1,840
of those, implicated in an incident:                6

Six incidents in a year from config, against two from code deploys — with config having none of the controls. That ratio is the argument.

Plus the incident-response improvement, measured on the next three incidents:

time to identify the cause when config was implicated
  before (config invisible on the timeline):  12 min median
  after:                                       2 min median

One honest limit: validation rules depend on facts about the environment (instanceCount, postgresMaxConnections) that can themselves change. We read them from the live environment at validation time rather than hardcoding, which means a validation passing today can be wrong after an autoscaling event — so the rollout abort remains the backstop. Static validation narrows the window; it does not close it.

What this generalizes to#

Classify changes by what they affect, not by how they are delivered. A behaviour change is a behaviour change whether it arrives as code, config, a database row, a DNS record, a feature flag or an IAM policy. Each delivery mechanism needs controls proportional to its blast radius, and the fast ones usually have the biggest.

Validation needs the context the author lacks. The error was arithmetic involving another system's limit. A rule that reads both numbers catches what a careful human and a reviewing human both missed — this is the class of check machines are strictly better at.

Progressive rollout is the control that works when the others fail. Review can miss it, validation can be incomplete, and a canary still limits the damage to 1/48th. Of the four fixes, this is the one that handles the bug nobody anticipated.

"What changed?" must have one answer. Splitting change history across systems means an incident responder starts by reconciling timelines. Emitting every production change onto one stream is a small piece of plumbing with an outsized effect on time-to-diagnose.

Related#

• The rollout with no off switch, discovered at 2am — when the revert path is the thing missing
• A feature flag read on four platforms needs one source of truth and four caches — flags as config
• The migration that could not be rolled back — another change class outside the code path

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.