We had alerting on failed requests: if an endpoint failed more than N times in a window, page the team. It had been in production for months. It had never fired.
That sounds like good news until you notice the endpoints had failed.
Why it was silent
The counter lived in process memory. The API ran on 8 workers. Every failure incremented the counter of whichever worker happened to serve the request — so the failures were spread across 8 separate counters, each of which saw roughly an eighth of the real number and never crossed the threshold.
Nothing was broken in the code you’d read. The bug was in an assumption: there is one process.
The fix
Move the count somewhere all workers share. I replaced it with a sliding window in Redis — each failure is written with its timestamp, entries older than the window are dropped, and the size of what’s left is the failure count across every worker.
Now when an endpoint degrades, the team is paged while it’s degrading, not when a user tells us.
What I took from it
- An alert that has never fired is not evidence of health. Fire it on purpose once.
- Any in-memory state in a multi-worker server is per-worker state, whether you meant it or not.