The dashboard nobody looked at, and the alert that fired every day
An alert that fires daily has trained everyone to ignore it, which is worse than no alert.
practices
42 posts
Your error message is an API, and changing it breaks someone
Somebody is grepping your log line, matching on your message, or alerting on its text.
The test that broke on every refactor was testing the wrong thing
A test coupled to call order and internal structure fails when nothing is broken.
The migration that could not be rolled back, and nobody noticed until it mattered
A column drop is irreversible the moment it runs. Expand-contract exists because deploys and migrations do not move together.
The retry that turned a blip into an outage
Synchronised retries from every client are a self-inflicted thundering herd. Jitter is the mechanism, not a detail.
The third time this filled up, I diagnosed it as an outage
A consumable with no recorded lifetime is indistinguishable from a service that is down.