Setting Up Monitoring Alerts That Don't Make You Want to Turn Off Your Phone

Bad monitoring alerts are almost worse than no monitoring alerts. When every minor blip sends a notification, you train yourself — and your team — to ignore them. And then when something actually breaks, you're several minutes or hours into ignoring the thing that mattered because the noise conditioned you to do so.

I've been on teams where the on-call engineer had their phone on silent because the alert volume was genuinely unbearable. That's a culture problem rooted in a technical misconfiguration, and it's more common than most organizations want to admit.

The Signal-to-Noise Problem

Good monitoring alerts require thoughtful calibration at the edges. The middle is easy — a site that's completely unreachable should obviously trigger an alert. The hard calls are:

— A server that's responding slowly but not down. Is that an alert?
— A check that failed once and then recovered. Alert?
— An SSL cert that expires in 45 days. Alert now?
— A 503 that lasted 90 seconds and then resolved itself. Did you wake someone up for that?

Every team answers these questions differently depending on the service's criticality. But there are patterns that work across contexts.

Require consecutive failures. Before firing a monitoring alert, most tools can be configured to wait for N consecutive check failures. Two or three consecutive failures means a transient network blip won't page you. A real outage will still trigger immediately because the failures will be sustained. This single setting eliminates probably 60% of false positive alerts on a typical monitoring setup.

Separate severity levels. Not all failures are equal. A warning (high latency, approaching thresholds) should go to a Slack channel. A critical failure (site down, SSL expired, database unreachable) should wake someone up. Routing these identically means either too much noise on the urgent channel or too little urgency overall. Neither is good.

Alert Routing That Actually Gets Acted On

The destination of your monitoring alerts matters as much as the content. Here's the routing model I've settled on after a lot of trial and error:

Slack/Teams channel (#alerts or #ops): All warnings, slow recoveries, SSL cert reminders at 30-day threshold, domain expiry notices 60+ days out. Things that are informational and can be reviewed during business hours. Not a phone notification — a channel people check when they're working.

Email: Similar to the Slack channel but useful as a backup and for people not in Slack. Also good for the weekly uptime summary report — a formatted digest of the past week's incidents and current status, not individual alerts.

SMS or push notification: Production downtime confirmed across multiple check locations. SSL expired (not just expiring — actually expired). Payment processing endpoint unreachable. These are the "someone needs to look at this now" scenarios, and this channel should fire rarely enough that it retains that urgency.

Phone call: Reserved for major incidents with significant business impact where SMS might be missed. Most teams don't need this for routine monitoring. If you're an on-call engineer for a service with a real SLA, it's worth having as a last-resort escalation.

Alertiva lets you configure these channels per monitor and per severity level, which is exactly the flexibility you need to avoid the "everything goes to one place at the same volume" problem.

The Maintenance Window You Keep Forgetting

One of the most common causes of alert fatigue is alerts firing during planned maintenance. You schedule a deployment, the site goes down for 3 minutes during a restart, and the monitoring system pages everyone. People learn to ignore those alerts — and then generalize that learning to other alerts.

Set maintenance windows. Literally every monitoring platform supports this. Before you do a deployment, add a 30-minute maintenance window. The system pauses alerts. The deployment runs. The window ends and monitoring resumes. Nothing gets paged, nothing gets ignored, and the on-call engineer doesn't have a minor heart attack every time someone pushes to production.

It takes 30 seconds to add a maintenance window. Make it part of your deployment checklist, not an afterthought.

The Escalation Model for Small Teams

Bigger organizations have PagerDuty and complex on-call rotations. For a two-person dev team or a solo developer, the escalation model is simpler but still worth thinking through.

Primary alert: goes to you immediately. Secondary alert: if the issue isn't acknowledged within 15 minutes, escalate to a second contact. This could be a colleague, a business owner, or even just a second email address you check from your phone. The point is that no critical alert gets silently lost because you were in a meeting or asleep.

Some monitoring alerts tools let you set up a team where alerts rotate. For a 3-person team, this spreads the on-call burden without requiring enterprise software. Alertiva's team features support this kind of configuration — not everything needs to land in your inbox every time.

Get the routing right, keep the noise low, and your monitoring alerts become genuinely useful infrastructure. Get it wrong and you've spent money to train your team to ignore problems. The difference is usually 20 minutes of thoughtful configuration at setup time.

---