An alert needs a decision attached.
Every page should answer a question: why are we interrupting someone right now?
Noise is a reliability risk
An alert that fires continuously without a clear action trains responders to treat the system as background noise. An alert that never fires until a service is unavailable fails in the opposite direction. A useful policy starts from user impact and a tested first action.
Use windows deliberately
Short windows detect fast failures but react to brief spikes. Longer windows smooth noise but can miss sharp incidents. Use a combination only when it reflects an actual need; document why each window exists.
Put context into the event
A useful page identifies the affected service, what is measured, the target and the runbook. It should separate unknown facts from observed outcomes, and provide a way to confirm recovery.
Remove recurring false positives
After an alert has fired repeatedly without requiring action, revisit its relevance. Improve the measurement, threshold or destination rather than leaving responders to suppress it manually.
Return to runbook methods →