API Monitoring & Alerting Template

Overview

Your API returns 200 OK on every health check while p95 latency quietly triples and a bad deploy pushes error rates from 0.02% to 0.8%. Nothing “fails” — but every consumer feels it. This is why uptime checks aren’t monitoring: services degrade before they die, and the only way to catch degradation is to measure the right indicators against explicit targets.

This template encodes that model — the same SLI/SLO/error-budget framework from the Google SRE book. It defines the indicators to measure (availability, latency, error rate, saturation), the objectives to promise, the alert thresholds that page a human, and the runbooks that tell the on-call engineer what to do next. Related resources: API Gateway for Microservices, Circuit Breaker Pattern, and Dependency Injection.

When to Use

Use this resource when:

• Launching a new API or version that needs uptime guarantees
• Auditing existing monitoring coverage after an incident
• Defining on-call alert rules and escalation policies
• Migrating from ad-hoc dashboards to SLO-driven alerting
• Onboarding a new on-call rotation that needs documented thresholds and runbooks

Skip it for prototypes still finding their API surface — premature SLOs get set on numbers that will be wrong in a month. For alternatives, see API Lifecycle Management Template.

Solution

The template below is the fill-in document your team produces once per API — metadata, indicators, targets, alert tiers, dashboard layout, and runbook links. Copy it into your docs system, replace the <placeholders>, and keep it next to the alert rules it describes.

# API Monitoring & Alerting: `<API Name>`

## 1. Service Metadata

| Field | Value |
|-------|-------|
| API Name | `name` |
| Owner Team | `@team-name` |
| Tier | `P0 (critical) / P1 (important) / P2 (standard)` |
| Consumer Count | Internal: X, External: Y |

## 2. SLIs (Indicators We Measure)

| SLI | Metric | Data Source |
|-----|--------|-------------|
| Availability | `% of requests returning 2xx/3xx` | Load balancer or gateway logs |
| Latency | `p95, p99 response time` | APM (Datadog, New Relic) |
| Error Rate | `% of 5xx responses / total` | Application logs |
| Throughput | `Requests per minute` | Metrics server (Prometheus) |
| Saturation | `CPU / Memory / DB connections` | Infrastructure metrics |

## 3. SLOs (Targets We Promise)

| SLO | Target | Measurement Window | Burn Rate Alert |
|-----|--------|--------------------|-----------------|
| Availability | 99.9% | 30 days | 2% budget in 1 hour |
| Latency p95 | < 200ms | 7 days | 5x normal in 1 hour |
| Error Rate | < 0.1% | 30 days | 10% budget in 1 day |

## 4. Alert Definitions

### 4.1. Page Alerts (Wake Someone Up)

| Condition | Threshold | Duration | Severity |
|-----------|-----------|----------|----------|
| Error rate > 1% | > 1% | 2 minutes | P1 |
| Latency p95 > 1s | > 1000ms | 3 minutes | P1 |
| Availability < 99% | < 99% | 1 minute | P0 |

### 4.2. Warning Alerts (Ticket / Slack)

| Condition | Threshold | Duration | Action |
|-----------|-----------|----------|--------|
| Error rate > 0.1% | > 0.1% | 10 minutes | Create Jira ticket |
| Latency p95 > 300ms | > 300ms | 15 minutes | Notify Slack channel |
| Traffic drop > 50% | < 50% baseline | 5 minutes | Page on-call (possible outage) |

### 4.3. Informational Alerts (Dashboard Only)

| Condition | Purpose |
|-----------|---------|
| Throughput > 10x baseline | Detect viral traffic or DDoS |
| 4xx rate > 5% | Detect client misconfiguration |

## 5. Dashboard Layout

**Row 1: Health Overview**
- Availability gauge (last 1h, 24h, 7d)
- Latency heatmap by endpoint
- Error rate timeline

**Row 2: Endpoint Breakdown**
- Top 10 endpoints by latency
- Top 10 endpoints by error rate
- Slowest traces (linked to APM)

**Row 3: Infrastructure**
- Pod/container CPU and memory
- Database connection pool
- Queue depth (if async)

## 6. Runbook Links

| Alert | Runbook |
|-------|---------|
| Error rate spike | `/runbooks/api-error-spike` |
| Latency degradation | `/runbooks/api-latency-spike` |
| Traffic drop | `/runbooks/api-traffic-drop` |

Explanation

SLIs are what you measure, SLOs are how good it must be, and alerts are when to act. The template separates page alerts (requires human intervention) from warnings (can wait for business hours). Burn rate alerts catch SLO violations early by tracking how fast your error budget is consumed. Dashboard rows group related metrics so on-call engineers can triage in under 30 seconds.

The tiering matters more than the numbers. A page means “a human must act within the hour” — anything less urgent that pages anyway is training the on-call rotation to ignore their phones. Warnings collect issues worth fixing this week: elevated error rates that self-healed, latency creeping toward the threshold. Informational signals exist for forensics — they explain the incident after the page fires, they don’t cause one. Most teams discover their real alert budget is about 2-3 pages per week; beyond that, response quality collapses regardless of how good the thresholds are.

Error budget math: an SLO of 99.9% gives you 0.1% of failed requests — at 1M requests/month that’s 1,000 bad requests, or ~43 minutes of downtime. A burn rate of 1 means you’ll exhaust the budget exactly at the window’s end; 14.4x means it burns in ~2 hours instead of 30 days. That’s why the SRE model uses two alert speeds: fast burn (1h window, 14.4x) pages immediately, and slow burn (6h window, 6x) files a ticket.

Choosing Thresholds

The numbers in the template are starting points, not gospel. Calibrate them against three inputs:

1. Baseline your SLIs first. Run the dashboard for 2-4 weeks before paging anyone. If your p95 naturally sits at 250ms, a 200ms alert threshold guarantees noise from day one — set the page threshold at ~2x baseline, the warning at ~1.3x.
2. Match the SLO to consumer pain, not to a round number. 99.9% is a default, not a requirement. An internal reporting API can live at 99% (7h/month budget); a payment API may need 99.95%. Ask: at what error rate do consumers actually file tickets? Set the SLO slightly above that line.
3. Pick duration to kill flapping. A 1-minute error spike that self-heals shouldn’t page. for: 2m on error rate and for: 3m on latency filters out single-burst anomalies while still catching real incidents within 5 minutes.

Multi-window burn rate alerting is the refinement most templates skip: pair a fast window (1h, pages) with a long window (6h-3d, tickets) so slow leaks — a 0.05% error drip that never spikes — still surface before the monthly budget dies.

Prometheus Alert Rules

Define alerts as code so they’re version-controlled and reviewable — the YAML below encodes the page tier from the template plus the fast burn rate rule. Each rule carries a runbook annotation that paging tools render as a link, which is what makes the alert actionable instead of just loud:

groups:
  - name: api_slo_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.01
        for: 2m
        labels:
          severity: P1
          team: platform
        annotations:
          summary: "Error rate above 1% for 2 minutes"
          runbook: "/runbooks/api-error-spike"

      - alert: HighLatencyP95
        expr: |
          histogram_quantile(0.95, rate(
            http_request_duration_seconds_bucket[5m]
          )) > 1.0
        for: 3m
        labels:
          severity: P1
          team: platform
        annotations:
          summary: "p95 latency above 1s for 3 minutes"
          runbook: "/runbooks/api-latency-spike"

      - alert: SLOBurnRateFast
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.002
        for: 5m
        labels:
          severity: P1
          team: platform
        annotations:
          summary: "SLO burn rate exceeds 2% budget in 1 hour"
          runbook: "/runbooks/slo-burn-rate"

      - alert: TrafficDrop
        expr: |
          sum(rate(http_requests_total[5m]))
          <
          sum(rate(http_requests_total[5m] offset 1h)) * 0.5
        for: 5m
        labels:
          severity: P1
          team: platform
        annotations:
          summary: "Traffic dropped 50% compared to 1 hour ago"
          runbook: "/runbooks/api-traffic-drop"

Grafana Dashboard JSON

A minimal dashboard panel for error rate tracking — import it under Dashboards → New → Import and adapt the metric names to your instrumentation (the example assumes the standard http_requests_total / http_request_duration_seconds naming from most OpenTelemetry and ingress exporters):

{
  "dashboard": {
    "title": "API Monitoring Overview",
    "panels": [
      {
        "title": "Error Rate (5xx)",
        "type": "stat",
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
            "legendFormat": "Error %"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "thresholds": {
              "steps": [
                { "value": null, "color": "green" },
                { "value": 0.1, "color": "yellow" },
                { "value": 1, "color": "red" }
              ]
            }
          }
        }
      },
      {
        "title": "p95 Latency by Endpoint",
        "type": "heatmap",
        "gridPos": { "h": 8, "w": 12, "x": 6, "y": 0 },
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum by (endpoint, le) (rate(http_request_duration_seconds_bucket[5m]))) * 1000",
            "legendFormat": "{{endpoint}}"
          }
        ],
        "fieldConfig": {
          "defaults": { "unit": "ms" }
        }
      }
    ]
  }
}

Runbook Template

Each alert must link to a runbook — the page that turns “something is wrong” into “here’s what to do about it” at 3 AM. A good runbook assumes the reader is asleep, stressed, and has never seen this alert before. Keep triage under 60 seconds, mitigation steps mechanical, and post-incident follow-ups explicit. Here is a minimal template:

# Runbook: API Error Spike

## Alert Condition
Error rate > 1% for 2+ minutes (P1)

## Quick Triage (under 60 seconds)
1. Check the dashboard: which endpoints are returning 5xx?
2. Check recent deployments: was there a release in the last 30 minutes?
3. Check dependency health: are any upstream services down?

## Mitigation Steps
1. If a bad deployment caused the spike, roll back to the previous version
2. If a dependency is down, enable circuit breaker fallback
3. If traffic is abnormal, enable rate limiting at the gateway

## Post-Incident
1. File an incident report within 24 hours
2. Add the root cause to the known issues list
3. Update this runbook with any new mitigation steps

Rolling It Out

Don’t enable every alert on day one — that’s how teams train themselves to ignore pages within a month. The sequence that works:

1. Week 1-2: measure only. Deploy the dashboard and the recording rules, no alerts. You’re building the baseline every threshold will be calibrated against.
2. Week 3: draft SLOs, ship warnings. Write the SLO table, get sign-off from the API owner, and enable the warning tier only — tickets and Slack, nothing that pages.
3. Week 4-6: tune. Watch the warning volume. If any rule fires more than ~3 times a week without action, the threshold is wrong, not the service. Adjust until the signal is clean.
4. Week 6+: enable paging. Turn on page alerts only after the warning tier has run quiet. Every page alert gets a runbook link before it goes live — an alert without a runbook is a 3 AM guessing game.

Keep the template itself under version control next to the alert rules. When the SLO changes, the rules change in the same commit — drift between documented targets and actual thresholds is how teams end up “meeting an SLO” nobody agreed to.

Variants

The template’s defaults assume a standard production API with real consumers. Adjust the shape to fit the context:

Context	Approach	Notes
Internal microservices	Lower SLOs, simpler alerts	99% availability, Slack-only alerts
Public SaaS API	Strict SLOs, multi-channel paging	99.99% availability, PagerDuty + SMS
Serverless / Lambda	Focus on cold start and concurrency	Alert on throttling, not CPU
Event-driven	Alert on lag and DLQ depth	Consumer lag is the equivalent of latency

Two more axes worth deciding explicitly. Single endpoint vs aggregate: critical paths (checkout, auth) deserve their own SLOs and their own alerts — a service-wide 99.9% can hide a checkout path failing 5% of the time. Business-hours vs 24/7: an internal tool nobody uses at 3 AM doesn’t need paging overnight; gate the channel by severity and schedule, not by convenience.

What Works

The practices below are the difference between an alerting system people trust and one that gets muted:

1. Alert on symptoms (latency, errors) not causes (disk full) to reduce noise
2. Set every alert threshold based on SLO burn rate, not arbitrary percentages
3. Include runbook links directly in alert messages
4. Review and tune alert thresholds monthly; false positives erode trust
5. Use different channels for page vs warning so on-call knows urgency immediately
6. Track alert volume per week to identify noisy alerts that need tuning
7. Add a “test alert” button in your alerting tool to verify paging works end-to-end
8. Write alerts as code (Prometheus YAML, Terraform) — reviewable, versioned, and reproducible across environments
9. Review SLOs quarterly with the API owner; targets drift as traffic and architecture change

Common Mistakes

And the failure modes that quietly destroy alerting systems:

1. Alerting on CPU > 80% without linking it to user-facing symptoms
2. Setting the same SLO for all APIs regardless of business criticality
3. Using mean latency instead of percentiles (means hide outliers)
4. Alerting on single errors without a duration or rate threshold
5. Forgetting to alert on traffic drops (absence of errors can mean total failure)
6. Not testing alert delivery (PagerDuty rotation, Slack webhook) before an incident
7. Creating alerts without runbooks, leaving on-call engineers to guess mitigation steps

Troubleshooting

• An alert that should have fired never did. Check the expression against raw metrics first — a label mismatch (status vs code, a renamed job) makes rate() return nothing, which reads as “zero errors”, not an error. Then check for: — a duration longer than the incident’s life swallows short outages entirely.
• Alerts fire constantly and nobody reacts. That’s alert fatigue, and the fix is subtraction, not tuning. Pull the last 30 days of firings; anything that resolved itself or required no action goes to dashboard-only. Keep paging for conditions that need a human within the hour.
• Pages arrive but Slack/PagerDuty delivery silently breaks. Webhooks expire, rotations end, tokens get revoked. Schedule a synthetic test alert weekly — if the test doesn’t reach a phone, nothing else will either.
• Metrics show gaps exactly when incidents happen. Scrape targets dying under load is a classic — your monitoring shares fate with the service it watches. Give Prometheus its own capacity headroom and alert on scrape failures as a first-class signal.
• p95 looks fine but users still complain. Percentiles on the wrong window hide short spikes — a 30-second storm is invisible in a 5-minute rate(). Add a short-window panel (1m) next to the SLO panels, or check whether the client-side timeout is lower than your p95 threshold.
• The SLO alert fires but the dashboard shows nothing wrong. Aggregation mismatch — the alert computes over sum(rate()) across endpoints while the panel breaks down per-endpoint. The aggregate can breach while every individual line looks healthy; drill into the expression’s actual grouping.

See Also

• API Performance Budget Template — the sibling doc that sets latency budgets before you alert on them
• API Error Handling Guideline — what those 5xx responses should look like when alerts fire
• Service Dependency Map Template — map upstreams before writing dependency alerts
• Circuit Breaker Pattern — the mitigation most runbooks point to
• Google SRE Book — Monitoring Distributed Systems — the source of the SLI/SLO model
• Google SRE Workbook — Alerting on SLOs — burn rate math and multi-window alerting
• Prometheus Alerting Rules — expression syntax reference
• Grafana Alerting — notification channels and silences

Companion code: monitoring companion resources — Prometheus rules file, Grafana dashboard JSON, and a fill-in runbook template.