Page on symptoms, not causes

PLACEHOLDER POST — seed data. Replace with your real writeup.

In one quarter the on-call rotation got 1,400 pages. Seven of them were real incidents, and people had learned to ignore the rest.

The rule we adopted

A page must mean a user is being hurt right now, and a human needs to act. Everything else goes to a ticket or a dashboard.

• CPU > 80%: a cause, not a page
• disk will fill in 4h: a ticket, unless it’s less than 30 minutes away
• checkout error budget burning 14x: page

How we got from 310 to 38

1. Exported every rule with how often it fired and whether anyone acted on it
2. Deleted every rule that fired more than 10 times with no action taken
3. Replaced per-host rules with per-service SLO burn-rate alerts

# multi-window burn rate: fast enough to catch outages, slow enough to ignore blips
expr: |
  (job:slo_errors:ratio_rate1h > 14.4 * 0.001)
  and
  (job:slo_errors:ratio_rate5m > 14.4 * 0.001)

The alerts we deleted weren’t wrong. They were answering questions nobody was asking at 3am.