The health check that returned 200 while the service was useless
Proving the process is running keeps a broken instance in the load balancer.
reliability
8 posts
The queue had no maximum, so the failure was memory and not latency
An unbounded buffer converts a throughput problem into an out-of-memory crash.
The webhook that arrived twice, and the order that shipped twice
At-least-once is the contract every provider offers. Your handler is where exactly-once gets built.
The lock held across a network call, and the deadlock that followed
A mutex held for microseconds is fine. Held across an unbounded wait, it serialises your service.
The rollout with no off switch, discovered at 2am
A flag that gates the new path but not the data migration is not a kill switch.
The config change that was really a deploy, and had none of the safeguards
Config bypasses code review, CI, canaries and rollback. It changes behaviour just as much.
Every timeout you set independently adds up to one that is wrong
Three services with 30-second timeouts give the user 90 seconds.
The retry that turned a blip into an outage
Synchronised retries from every client are a self-inflicted thundering herd. Jitter is the mechanism, not a detail.