On-Call Runbook Template

Overview

At 3 a.m., a junior engineer receives a page: “Database connection pool exhausted.” Without one, they spend 30 minutes Googling instead of 5 minutes following a checklist. A runbook isn’t a luxury for large teams — it’s a survival tool for whoever is on-call. This template structures common alerts, diagnostic steps, and resolution procedures so whoever is on-call acts with confidence instead of fear.

When to Use

Use this resource when:

• You’re creating your team’s first on-call rotation and have no documented procedures
• Your mean-time-to-resolution (MTTR) stays high because engineers debug from scratch every time
• You are onboarding new team members who will join the on-call rotation

Solution

# On-Call Runbook: `<Service / Team>`

## 1. Alert Index

| Alert Name | Severity | Page? | Runbook Section | Last Verified |
|------------|----------|-------|-----------------|---------------|
| High Error Rate | SEV 2 | Yes | 2.1 | `YYYY-MM-DD` |
| Latency P99 > 2s | SEV 2 | Yes | 2.2 | `YYYY-MM-DD` |
| Disk Usage > 85% | SEV 3 | No | 2.3 | `YYYY-MM-DD` |
| Memory Usage > 90% | SEV 3 | No | 2.4 | `YYYY-MM-DD` |
| SSL Expiry < 7 days | SEV 3 | No | 2.5 | `YYYY-MM-DD` |
| Dependency Unhealthy | SEV 2 | Yes | 2.6 | `YYYY-MM-DD` |
| Job Queue Backlog | SEV 3 | No | 2.7 | `YYYY-MM-DD` |

## 2. Response Procedures

### 2.1. High Error Rate

**Symptoms:**
- Error rate > 1% (or threshold defined in alert)
- Spike in 5xx responses

**Diagnostic Steps:**
1. Check error dashboard for top error types
2. Correlate with recent deployments (last 2 hours)
3. Check downstream dependency health
4. Review application logs for stack traces

**Resolution:**
- If caused by deployment: execute rollback plan
- If caused by dependency failure: see 2.6 Dependency Unhealthy
- If caused by resource exhaustion: see 2.3 or 2.4
- If transient spike: monitor for 10 minutes; auto-recovery common

**Escalation:**
- If error rate > 10% or data-loss errors: page team lead (SEV 1)
- If no root cause within 30 minutes: page team lead

### 2.2. Latency P99 > 2s

**Symptoms:**
- P99 latency above threshold
- User complaints about slow responses

**Diagnostic Steps:**
1. Check database query latency
2. Check cache hit rate (Redis / Memcached)
3. Check for N+1 query patterns in logs
4. Check downstream service latency
5. Review CPU and memory utilization

**Resolution:**
- If database bottleneck: kill long-running queries, scale read replicas
- If cache miss storm: pre-warm cache, increase TTL temporarily
- If downstream latency: see 2.6 Dependency Unhealthy

**Escalation:**
- If latency > 10s or affecting > 50% of users: page team lead
- If caused by DDoS: engage security team immediately

### 2.3. Disk Usage > 85%

**Symptoms:**
- Disk usage alert firing
- Risk of write failures

**Diagnostic Steps:**
1. Identify largest directories (`du -sh /* | sort -rh | head`)
2. Check log rotation configuration
3. Check for temporary files or core dumps
4. Check database size and growth rate

**Resolution:**
- Clean old logs (ensure retention policy allows)
- Truncate oversized tables / partitions
- Expand disk if cloud-hosted (AWS EBS, GCP PD)
- Enable log rotation if disabled

**Escalation:**
- If > 95% and writes failing: page team lead
- If expansion fails: page infrastructure team

### 2.4. Memory Usage > 90%

**Symptoms:**
- Memory usage alert firing
- Risk of OOM kills

**Diagnostic Steps:**
1. Identify memory-hungry processes (`ps aux --sort=-%mem | head`)
2. Check for memory leaks (trend over 7 days)
3. Check cache size and eviction rate
4. Check for unbounded queue growth

**Resolution:**
- Restart service if leak suspected (temporary fix)
- Scale to larger instance if sustained growth
- Reduce cache size or TTL
- Fix code leak in next release

**Escalation:**
- If OOM kills causing restarts: page team lead
- If leak root cause unclear: page team lead

### 2.5. SSL Expiry < 7 Days

**Symptoms:**
- Certificate expiration warning

**Diagnostic Steps:**
1. Confirm certificate details and exact expiry date
2. Verify auto-renewal is configured
3. Check if cert is deployed on all endpoints

**Resolution:**
- If auto-renewal failed: manually renew (see cert runbook)
- If manual cert: create renewal ticket for SRE team
- Deploy renewed cert to all load balancers / CDNs

**Escalation:**
- If expiry < 24 hours: page SRE team lead

### 2.6. Dependency Unhealthy

**Symptoms:**
- Downstream service health check failing
- Timeout errors to specific endpoint

**Diagnostic Steps:**
1. Check dependency status page
2. Check dependency metrics dashboard
3. Verify network connectivity (ping, traceroute)
4. Check for DNS resolution issues
5. Verify authentication tokens / API keys not expired

**Resolution:**
- If dependency outage: enable circuit breaker, serve degraded mode
- If network issue: engage network / cloud provider
- If auth issue: rotate credentials
- If capacity issue: request dependency team scale

**Escalation:**
- If dependency is critical and no degraded mode: page team lead + dependency team

### 2.7. Job Queue Backlog

**Symptoms:**
- Queue depth growing
- Processing lag increasing

**Diagnostic Steps:**
1. Check worker process count and health
2. Check worker CPU / memory utilization
3. Check for dead-letter queue growth
4. Review job failure rate

**Resolution:**
- Scale workers horizontally if CPU < 70%
- Restart stuck workers
- Retry failed jobs from dead-letter queue
- If database bottleneck: scale read replicas

**Escalation:**
- If backlog > 1 hour and growing: page team lead

Explanation

The runbook treats every alert as a diagnostic workflow, not a problem to brute-force. By forcing the engineer to check specific things in order, it cuts the chance of misdiagnosis — restarting a service when the real issue is a downstream dependency is the classic mistake. The escalation rules keep the on-call engineer from going quiet for hours while they struggle alone.

How to Build and Maintain the Runbook

A runbook that doesn’t match your alerts is worse than none — it trains engineers to ignore it. Build yours from evidence, not imagination.

1. Mine your incident history. Pull the last 6–12 months of pages and tickets. Group them by root cause, not by alert name — three different alerts often share one fix. The seven procedures in the template cover the usual suspects; your own history tells you which to write first.

2. Map severity to channel before you need it. Decide upfront what pages a human and what files a ticket. A simple map keeps alert fatigue from eating the rotation:

Severity	Channel	Response expectation
SEV 1 — user-facing outage or data loss	Page + incident channel	Acknowledge in 5 min
SEV 2 — degraded or at-risk	Page	Acknowledge in 15 min
SEV 3 — warning, no user impact yet	Ticket / dashboard	Triage next business day

3. Write for the person who has never seen the alert. Exact commands over descriptions (“run kubectl top pods”, not “check resource usage”). If a step needs judgment, say what “bad” looks like (“hit rate under 80% is your problem”).

4. Wire alerts to runbook sections. Use the runbook: annotation shown below so the page itself carries the link. Nobody should have to search for the doc at 3 a.m. — the monitoring and alerting guide covers the annotation pattern in depth, and the on-call incident response guide covers the rotation workflow around it.

5. Close the loop after every incident. The post-incident checklist at the end of this page is the maintenance mechanism: new alert → new runbook entry, missed diagnosis → corrected steps. Then verify every entry quarterly — the “Last Verified” column in the alert index exists so stale procedures get caught before the next 3 a.m. page does.

6. Assign an owner. A runbook with no owner rots. One engineer owns it (usually the on-call lead), and “update the runbook” is part of every incident’s definition of done — see the prometheus alerting recipe for wiring the alert side of that loop.

Real-World Runbook Lessons

Runbooks earn their keep in the incidents everyone remembers — and in the small ones nobody does.

Knight Capital (2012). A deployment reused a flag that woke up code dormant since 2003, and the firm lost $440M in 45 minutes. There was no documented procedure for the flag, the kill, or the rollback — by the time anyone understood what the system was doing, the market had taken the money. The classic argument for “every deploy procedure has a written undo step.”

AWS S3 outage (2017). An operator debugging a slow billing system ran a command that removed more capacity than intended — and took down a huge slice of the internet for hours. Amazon’s own postmortem said the fix included “adding safeguards to prevent capacity from being removed below a minimum level” — a constraint that belonged in the runbook step, not in the operator’s head.

GitLab database incident (2017). Working through replication problems at night, an engineer deleted the production database directory and lost six hours of data. GitLab’s famously transparent postmortem is itself a runbook artifact: they documented exactly what happened and what would have helped — the alert fatigue and missing safeguards that a live runbook captures.

Netflix gamedays. Netflix runs deliberate failure drills (the DiRT exercises that grew into Chaos Engineering) partly to keep runbooks honest — an untested procedure is fiction until someone follows it under pressure. That’s what the “Last Verified” column in the alert index is for: if the date is stale, the procedure is a hypothesis, not a plan.

The common thread: incidents punish the gap between “we know what to do” and “it’s written, tested, and linked from the alert.”

Variants

Context	Alert Focus	Key Addition
Kubernetes	Pod restarts, node pressure, ingress errors	kubectl commands for pod inspection
Serverless	Lambda errors, cold starts, throttling	CloudWatch Logs Insights queries
Mobile backend	Push notification failures, API rate limits	Device-specific error segmentation
Data pipeline	Job failures, schema drift, late data	Airflow / Dagster task retry procedures
Multi-region	Region-specific latency, replication lag	Failover runbook section

What Works

1. Keep each procedure under one page; long runbooks are not read during incidents
2. Include exact commands, not just “check logs”; stress reduces typing accuracy
3. Verify every procedure quarterly; stale runbooks are worse than none
4. Link to dashboards and logs, do not paste screenshots that go stale
5. Include a “when to escalate” decision for every alert; ambiguity causes delay

Common Mistakes

1. Writing runbooks for experts; they are for the engineer who has never seen this alert
2. Not testing runbook commands in a staging environment
3. Omitting rollback steps; sometimes the fix is “undo the last change”
4. Creating runbooks but not linking them from the alerting system
5. Treating runbooks as static documents; they must be updated after every incident

Advanced Solutions

Automated runbook execution with diagnostic scripts

Pre-wire common diagnostic steps into executable scripts that the on-call engineer can run with a single command:

#!/bin/bash
# diagnose.sh - Automated diagnostic collector for on-call engineers
# Usage: ./diagnose.sh <service-name>

set -euo pipefail

SERVICE="${1:?Usage: $0 <service-name>}"
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
REPORT_DIR="/tmp/diagnostics-${SERVICE}-${TIMESTAMP}"

mkdir -p "$REPORT_DIR"

echo "=== Collecting diagnostics for $SERVICE at $TIMESTAMP ==="

# 1. Service status
echo "Checking service status..."
systemctl status "$SERVICE" 2>&1 | tee "$REPORT_DIR/service-status.txt" || true

# 2. Recent logs (last 100 lines)
echo "Collecting recent logs..."
journalctl -u "$SERVICE" --since "1 hour ago" --no-pager 2>&1 \
  | tail -100 > "$REPORT_DIR/recent-logs.txt" || true

# 3. Resource utilization
echo "Checking resource utilization..."
{
  echo "=== CPU ==="
  top -bn1 | head -20
  echo ""
  echo "=== Memory ==="
  free -h
  echo ""
  echo "=== Disk ==="
  df -h
  echo ""
  echo "=== Top processes by CPU ==="
  ps aux --sort=-%cpu | head -10
  echo ""
  echo "=== Top processes by Memory ==="
  ps aux --sort=-%mem | head -10
} > "$REPORT_DIR/resources.txt"

# 4. Network connectivity
echo "Checking network..."
{
  echo "=== Listening ports ==="
  ss -tlnp 2>/dev/null || netstat -tlnp 2>/dev/null
  echo ""
  echo "=== Active connections ==="
  ss -tn state established 2>/dev/null | head -20
} > "$REPORT_DIR/network.txt"

# 5. Recent deployments
echo "Checking recent deployments..."
{
  echo "=== Docker containers ==="
  docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || echo "Docker not available"
  echo ""
  echo "=== Kubernetes pods ==="
  kubectl get pods -l app="$SERVICE" 2>/dev/null || echo "kubectl not available or no pods found"
} > "$REPORT_DIR/deployments.txt"

# 6. Health check
echo "Running health check..."
curl -sS -o "$REPORT_DIR/health-response.txt" -w "%{http_code}" \
  "http://localhost:8080/health" 2>&1 | tee "$REPORT_DIR/health-status.txt" || true

echo ""
echo "=== Diagnostics complete ==="
echo "Report saved to: $REPORT_DIR"
echo "Review files and attach to incident ticket."

Kubernetes-specific diagnostic commands

For containerized environments, include kubectl one-liners the on-call engineer can copy-paste:

# Quick pod status check
kubectl get pods -n production -o wide | grep -v Running

# Get logs from a crashing pod
kubectl logs -n production <pod-name> --previous --tail=50

# Describe a pod for events and conditions
kubectl describe pod -n production <pod-name>

# Check resource usage across nodes
kubectl top nodes
kubectl top pods -n production --sort-by=memory

# Execute into a pod for network debugging
kubectl exec -it -n production <pod-name> -- /bin/sh -c "nslookup <dependency>"

# Check recent events in namespace
kubectl get events -n production --sort-by='.lastTimestamp' | tail -20

# Port-forward for local debugging
kubectl port-forward -n production svc/<service-name> 8080:80

Alert-to-runbook linking with Prometheus annotations

Link alerts directly to runbook sections using Prometheus alert labels so nobody has to hunt for the right procedure mid-incident:

# prometheus/alerts.yml
groups:
  - name: service-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) > 0.01
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.service }}"
          runbook: "https://wiki.internal/runbooks/on-call#21-high-error-rate"
          dashboard: "https://grafana.internal/d/service-overview?var-service={{ $labels.service }}"

      - alert: DiskUsageHigh
        expr: |
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 10m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Disk usage > 85% on {{ $labels.instance }}"
          description: "Disk usage is {{ $value }}% on {{ $labels.instance }}"
          runbook: "https://wiki.internal/runbooks/on-call#23-disk-usage--85"
          dashboard: "https://grafana.internal/d/node-overview?var-node={{ $labels.instance }}"

Post-incident runbook update checklist

After every incident, verify the runbook picked up the lessons learned:

## Post-Incident Runbook Update

- [ ] Was the alert in the runbook? If no, add it now
- [ ] Were the diagnostic steps accurate? Update if they missed the root cause
- [ ] Were the resolution steps correct? Update if they did not work
- [ ] Was the escalation threshold appropriate? Adjust if too high or too low
- [ ] Did the runbook link from the alert work? Fix if broken
- [ ] Are there new commands that would have helped? Add them
- [ ] Was the "last verified" date updated? Set to today
- [ ] Did the on-call engineer find the runbook useful? Note feedback

## Anti-patterns — 2.1 High Error Rate

- Do NOT restart all pods simultaneously (causes cascading failures)
- Do NOT scale up without checking if the issue is downstream
- Do NOT deploy a fix without testing in staging first
- Do NOT close the alert until error rate is below threshold for 15 minutes

See Also

• Google SRE Book — Being On-Call: Google’s chapter on structuring on-call rotations and what belongs in a runbook
• Google SRE Workbook: the practical companion with runbook examples from real SRE teams
• PagerDuty Incident Response: the incident response framework this template’s severity map borrows from
• Atlassian Incident Management Handbook: a solid reference for escalation paths and post-incident reviews
• On-call incident response guide: the rotation workflow around this runbook
• Monitoring and alerting guide: how to wire the alert-to-runbook annotations

Companion code: on-call runbook resources: the runbook template, diagnose.sh, the Prometheus alerts file, the kubectl diagnostics cheatsheet, and the post-incident checklist from this page.