Monitoring and Alerting Policy Template

Overview

A Monitoring and Alerting Policy defines how an organization detects problems, notifies the right people, and escalates when issues aren’t resolved quickly. Without a clear policy, teams drift into one of two failure modes: alert fatigue from paging on noise, or missed incidents because nobody owned the signal. This template gives you a structured framework for severity levels, thresholds, routing rules, escalation paths, and the review cadence that keeps the whole thing honest.

A typical trigger for writing this policy: the on-call rotation just burned out because P2 pages fired all night on alerts nobody could act on. Writing the rules down — who gets paged, for what, and when it escalates — is the fix. If you need the monitoring platform itself rather than the policy layer, the Complete Guide to Observability with the Grafana Stack covers the tooling side.

When to Use

• Setting up a new observability platform or monitoring stack.
• Onboarding a new service or team into the alerting system.
• Reviewing alert quality after a period of noise or missed incidents.
• Defining on-call responsibilities and escalation paths.
• Preparing for an audit of operational maturity or incident response.

When not to use it: a two-person team running one service doesn’t need a formal routing matrix — a shared Slack channel and a paging rotation will do. The policy earns its keep once two or three teams share infrastructure and “who responds” stops being obvious.

Prerequisites

• A monitoring and observability platform such as Prometheus, Datadog, Grafana, New Relic, or PagerDuty.
• A list of critical services and infrastructure components.
• Defined on-call rotations and escalation contacts.
• A communication channel for alerts, such as Slack, Microsoft Teams, or email.
• An incident response process that alerts will trigger.

The Policy Template

The six tables below are the template body. Copy them into your internal docs, replace the example values, and fill in your own teams, channels, and thresholds. The prose after each table explains the reasoning so the policy survives contact with reviewers.

1. Alert Severity Levels

Severity	Response Time	Example	Notification Channel
P1 - Critical	Immediate (5 min)	Service down, data loss, revenue impact	Page on-call + executive notification
P2 - High	15 minutes	Degraded performance, failed backups	Page on-call + Slack alert
P3 - Medium	1 hour	High error rate, resource pressure	Slack or email to owning team
P4 - Low	Next business day	Capacity warning, non-urgent drift	Email or dashboard notice
P5 - Informational	None	Usage metrics, trend data	Dashboard only

The important line is between P2 and P3: P1 and P2 interrupt a human, everything below doesn’t. Keep that line strict. If you find yourself paging on P3 events, either they’re miscategorized or your thresholds are too tight — the scorecard below helps you catch that.

2. Alert Categories

Category	Purpose	Examples
Availability	Detect service unreachability	HTTP 5xx, connection timeout, health check failure
Performance	Detect latency and throughput issues	p99 latency > 500ms, queue depth high
Capacity	Detect resource exhaustion	CPU > 85%, disk > 80%, memory pressure
Error rate	Detect unusual failure rates	Error rate > 1% for 5 minutes
Security	Detect suspicious activity	Failed logins, rate limit hits, blocked traffic
Business	Detect revenue or workflow impact	Failed payments, order drop, signup failure
Data health	Detect pipeline or data quality issues	Stale data, missing partitions, sync lag

Categories matter for routing more than for dashboards. An availability alert and a business alert often need different responders even when they fire on the same service, so label every rule with its category and let the routing layer sort it out.

3. Alert Routing Matrix

Team	Primary Hours	On-Call Hours	Channels	Escalation Path
Platform team	08:00 - 18:00 UTC	24/7	PagerDuty, #platform-alerts	Manager, then VP Engineering
Application team	08:00 - 18:00 UTC	24/7	PagerDuty, #app-alerts	Team lead, then Engineering manager
Security team	24/7	24/7	PagerDuty, #security-alerts	Security lead, then CISO
Database team	08:00 - 18:00 UTC	24/7	PagerDuty, #db-alerts	DBA lead, then Platform manager
Business operations	Business hours	None	Email, Slack	Operations manager

Route to the team that can fix the problem, not to a central queue that re-dispatches. Every hop between the alert and the responder adds minutes you can’t afford on a P1. The escalation path column pairs naturally with an escalation policy template if you need the full on-call contract.

4. Alert Threshold Guidelines

Signal	Warning Threshold	Critical Threshold	Evaluation Window
HTTP error rate	> 1% for 5 min	> 5% for 2 min	Rolling 5 min
Response latency p99	> 500ms for 10 min	> 1s for 5 min	Rolling 10 min
CPU utilization	> 70% for 10 min	> 90% for 5 min	Rolling 5 min
Disk utilization	> 75% for 1 hour	> 90% for 15 min	Rolling 15 min
Memory utilization	> 80% for 10 min	> 95% for 5 min	Rolling 5 min
Queue depth	> 1000 for 10 min	> 5000 for 5 min	Rolling 5 min
Failed backup	N/A	Any failed backup	Per job run
SSL certificate expiry	< 30 days	< 7 days	Daily check

Treat these numbers as starting points, not gospel. Pull your last 90 days of metrics, look at where real incidents started, and set the warning threshold just below that line. Thresholds copied from a template without historical data are how you end up paging on noise — and the quarterly calibration row in the maintenance table exists to fix exactly that.

5. Escalation Rules

Severity	Initial Alert	No Acknowledgment	Still Unresolved	Final Escalation
P1	Page on-call immediately	5 min	15 min	Executive notification + war room
P2	Page on-call	15 min	30 min	Manager page
P3	Slack to owning team	1 hour	4 hours	Manager notification
P4	Email or dashboard	Next business day	N/A	Weekly review

Escalation timers are a contract with your on-call engineers: a page that isn’t acknowledged gets a louder page, not silence. If people routinely ignore the first notification, that’s a signal about alert quality, not about the engineers.

6. Alert Review and Maintenance

Activity	Frequency	Owner	Output
Alert quality review	Weekly	On-call engineer	Top noisy alerts, tuning actions
Alert runbook review	Monthly	SRE team	Updated runbooks for each alert
Threshold calibration	Quarterly	Observability team	Threshold adjustments with evidence
On-call retro	After major incident	Incident commander	Alert improvements, follow-up tasks
Policy review	Annually	Engineering leadership	Updated policy document

The weekly quality review is the most valuable row in this table. Fifteen minutes of “which alerts fired, which were actionable” catches alert fatigue before it becomes a staffing problem. Post-incident reviews pair well with a postmortem incident review template when the incident is big enough to document.

How the Policy Works in Practice

An alert follows a predictable lifecycle: a metric crosses a threshold, the rule fires, the routing layer hands it to the owning team, someone acknowledges it, and the escalation timer runs in the background in case they don’t. The review loop at the end is what keeps the system calibrated — every noisy alert that survives a review becomes tomorrow’s missed page.

Two design choices carry most of the weight. First, severity drives both the channel and the clock: paging channels exist only for P1/P2, and escalation timers shrink as severity rises. Second, every alert carries a runbook link, so the responder’s first action is documented rather than improvised.

Prometheus Alert Rules Example

Here’s how the policy’s severity labels and thresholds translate into real Prometheus rules — each alert carries severity and team labels that the routing config below keys on:

groups:
  - name: api_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 2m
        labels:
          severity: P1
          team: platform
        annotations:
          summary: "{{ $labels.service }} error rate > 5%"
          description: "{{ $labels.service }} has {{ $value | humanizePercentage }} error rate for 2 minutes"
          runbook: "https://runbooks.example.com/high-error-rate"

      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[10m])) by (le, service)) > 1
        for: 5m
        labels:
          severity: P2
          team: platform
        annotations:
          summary: "{{ $labels.service }} p99 latency > 1s"
          runbook: "https://runbooks.example.com/high-latency"

      - alert: DiskSpaceLow
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"}
          / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 15m
        labels:
          severity: P2
          team: infrastructure
        annotations:
          summary: "Disk space < 10% on {{ $labels.instance }}"
          runbook: "https://runbooks.example.com/disk-space"

Alertmanager Routing Configuration

The routing tree mirrors the policy’s severity table: P1/P2 go to PagerDuty with aggressive repeat intervals, P3 goes to Slack, P4 to email. Everything else falls through to the default receiver.

route:
  receiver: default
  group_by: ["alertname", "service", "severity"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "P1"
      receiver: pagerduty-critical
      group_wait: 0s
      repeat_interval: 30m
    - matchers:
        - severity = "P2"
      receiver: pagerduty-warning
      group_wait: 30s
      repeat_interval: 2h
    - matchers:
        - severity = "P3"
      receiver: slack-alerts
      group_wait: 5m
    - matchers:
        - severity = "P4"
      receiver: email-alerts
      group_wait: 1h

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: "P1_KEY"
  - name: pagerduty-warning
    pagerduty_configs:
      - service_key: "P2_KEY"
  - name: slack-alerts
    slack_configs:
      - channel: "#alerts"
        api_url: "SLACK_WEBHOOK_URL"
  - name: email-alerts
    email_configs:
      - to: "team@example.com"
  - name: default
    slack_configs:
      - channel: "#alerts"

group_wait and repeat_interval encode the escalation contract in config: P1 alerts group instantly and re-notify every 30 minutes until acknowledged, while P4 batches hourly. Tune these before you tune thresholds — grouping is usually the cheapest way to cut page volume.

Alert Quality Scorecard

Review each alert quarterly using this scorecard — six criteria, 1–5 points each, 30 points maximum:

Criterion	Score (1-5)	Notes
Actionable: Does the alert trigger a clear response?
Accuracy: Is the false-positive rate below 5%?
Timely: Does the alert fire before user impact?
Routed: Does it reach the team that can fix it?
Documented: Is there a runbook linked?
Unique: Is this alert redundant with another?

Score interpretation: 18 or more means the alert is healthy. Between 12 and 17, put it on a 30-day improvement plan with a named owner — usually that means tightening the threshold, fixing the routing, or writing the missing runbook. Below 12, delete it; an alert that isn’t actionable, accurate, or routed is training your team to ignore pages.

How to Roll Out This Policy

Adopting the template works best as a staged rollout rather than a big-bang rewrite:

1. Inventory — list every existing alert, its severity claim, and where it routes. Most teams discover orphan alerts going to dead channels at this step.
2. Draft severities — map existing alerts to the P1–P5 table. Expect most “critical” alerts to land at P3; that’s normal, not a problem.
3. Wire routing — configure the routing matrix in your tooling (the Alertmanager config above covers the Prometheus stack) and test each path with a synthetic alert.
4. Pilot on one team — run the policy with a single team for two weeks. Track pages per shift and false-positive rate.
5. Review and widen — tune thresholds from the pilot data, then roll out to the remaining teams and start the weekly quality review.

The two-week pilot matters more than the document. A policy that survives contact with real alerts earns trust; one that pages at 3 a.m. for a capacity warning gets routed around within a month.

One last ownership note: assign the document itself an owner — usually the SRE or platform lead — and keep it in version control next to the alert configs it describes. Changes to thresholds or routing should go through the same pull-request review as the config changes that implement them. A policy that lives in a wiki nobody watches is indistinguishable from no policy at all.

Variants

• Cloud-native alerting policy: Uses Prometheus Alertmanager, Grafana OnCall, or PagerDuty for container and serverless environments.
• Enterprise IT monitoring policy: Focuses on infrastructure, network, and service desk integration.
• Security alerting policy: Emphasizes SIEM rules, threat detection, and incident response triggers.
• Business operations alerting: Tracks KPIs, revenue, and customer-facing metrics with business-hour notifications.
• Developer self-service alerting: Allows teams to define their own alert rules within guardrails.
• Multi-cloud alerting policy: Uses cloud-agnostic tools (Grafana, Datadog) to unify alerts across AWS, GCP, and Azure.
• Cost alerting policy: Monitors cloud spend with budget thresholds and anomaly detection on cost metrics.
• Compliance alerting policy: Tracks audit log gaps, failed access reviews, and policy violations for regulated environments.

What Works

• Alert on symptoms that affect users, not just internal metrics.
• Use multi-window or multi-burn-rate thresholds to reduce false positives.
• Require every alert to have an associated runbook or troubleshooting link.
• Route alerts to the team that can fix the problem, not a central queue.
• Keep alert messages concise and include context such as severity, service, and impact.
• Review noisy alerts weekly and tune or delete them.
• Test escalation paths during regular drills.
• Document alert thresholds and the rationale for changes.
• Use alert grouping to batch related alerts into a single notification during incidents.
• Implement alert suppression for known maintenance windows and deployments.

Common Mistakes

• Alerting on every metric threshold without considering user impact.
• Sending all alerts to a single channel with no routing.
• Using the same severity for every alert.
• Not requiring acknowledgment or tracking resolution time.
• Ignoring alerts that fire repeatedly without action.
• Missing escalation paths for severe incidents.
• Failing to review and retire stale alerts after system changes.
• Setting thresholds based on gut feeling instead of historical data.
• Not including service name and environment in alert labels, making triage harder.
• Alerting on absolute values instead of rates or ratios, causing false positives during traffic spikes.

Troubleshooting

• Alert storm during a deploy: suppress alerts for the affected services during the window, or tighten group_by so 50 related alerts collapse into one notification.
• Threshold flapping (fires, resolves, fires again): widen the evaluation window or add a second condition — a for clause alone won’t stop a metric oscillating around the line.
• Pages for alerts nobody can act on: score them with the scorecard above; anything below 12 gets deleted, not silenced.
• Alert fires but there’s no runbook: the annotation was skipped. Make the runbook link a required field in code review for new alert rules.
• A silence left active after maintenance: expired silences are easy to forget; review active silences in the weekly alert review so yesterday’s maintenance doesn’t mute today’s incident.
• Same alert routed to three channels: deduplicate receivers in the routing config, or accept it — but page duplication is a fast path to alert fatigue.

Further Reading

• Google SRE Book — Monitoring Distributed Systems and the Alerting on SLOs workbook chapter
• Prometheus documentation: Alerting rules
• Alertmanager documentation: Configuration
• PagerDuty: Alert management best practices
• Related on this site: the SLO template for error-budget-based alerting.