Postmortem Incident Review Template

Overview

Every outage is a lesson that someone will repeat unless it is written down. Postmortems are not about blame — they are about understanding how a system with good people and good intentions still failed. A well-run postmortem reconstructs what happened, identifies the chain of events that led to failure, and produces concrete actions that make the next incident less likely or less severe.

When to Use

• For alternatives, see Blameless Postmortems: Learning from Incidents Without Blame.

Use this template when:

• A service-impacting incident has been resolved
• An incident exceeded a severity threshold (e.g., SEV-2 or higher)
• An incident caused data loss, security exposure, or compliance impact
• A near-miss revealed a major risk that did not materialize
• A recurring issue suggests a deeper systemic problem

Prerequisites

Before running a postmortem:

• The incident is fully resolved and systems are stable
• A timeline of events has been collected (see incident timeline template)
• Key participants are available: responders, engineers involved, and observers
• Leadership supports a blameless process
• There is a decision on whether the postmortem is internal-only or customer-facing

Solution

# Postmortem: `<Incident Title>`

> Incident ID: ______ | Date: ______ | Severity: ______
> Lead responder: ______ | Postmortem owner: ______
> Review date: ______ | Status: Draft / Reviewed / Approved

## 1. Executive Summary

- **What happened:** ______
- **Impact:** ______
- **Duration:** ______
- **Root cause (one sentence):** ______
- **Status:** ______

## 2. Impact Assessment

| Metric | Value |
|--------|-------|
| Services affected | ______ |
| Users affected | ______ |
| Error rate increase | ______ |
| Revenue / transaction impact | ______ |
| Data affected | ______ |
| SLA / SLO impact | ______ |

## 3. Timeline

| Time (UTC) | Event | Source |
|------------|-------|--------|
| ______ | ______ | ______ |
| ______ | ______ | ______ |
| ______ | ______ | ______ |

## 4. Root Cause Analysis

### What was the trigger?

______

### What was the contributing factor?

______

### Why did detection take longer than expected?

______

### Why did recovery take longer than expected?

______

### What defenses failed or were missing?

______

## 5. Lessons Learned

### What went well

- ______
- ______

### What went wrong

- ______
- ______

### Where we got lucky

- ______
- ______

## 6. Action Items

| Action | Owner | Due Date | Priority | Status |
|--------|-------|----------|----------|--------|
| ______ | ______ | ______ | P0 / P1 / P2 | ______ |

## 7. Communication

- [ ] Internal stakeholders notified
- [ ] Customer-facing post published (if applicable)
- [ ] Support team briefed
- [ ] Status page updated with resolution

## 8. Appendix

- Links to dashboards: ______
- Links to logs: ______
- Links to incident channels: ______
- Related tickets: ______

Explanation

The template separates the story (timeline, what happened) from the analysis (why it happened) from the action (what we will do). The root cause analysis section uses a chain of questions that expose not just the trigger but the conditions that allowed the trigger to cause an outage. The “where we got lucky” section is critical: it identifies near-misses and hidden risks that did not materialize this time but may next time.

Real Postmortem Example

# Postmortem: INC-2026-07-11-001 — Login Failures in EU

## Summary
On July 11, 2026, the authentication service experienced a 15%
error rate for login attempts in the EU region for 30 minutes.
The root cause was a configuration change that altered the JWT
secret rotation interval, causing token validation to fail for
active sessions.

## Impact
- Duration: 30 minutes (10:55 - 11:25 UTC)
- Users affected: ~15,000 (15% of login attempts)
- Region: eu-west-1
- Revenue lost: estimated $2,500
- Support tickets: 47

## Timeline
- 10:42  DB CPU begins rising
- 10:55  PagerDuty alerts fire
- 11:00  SEV1 declared
- 11:03  Config change identified as cause
- 11:05  Rollback initiated
- 11:08  Rollback deployed
- 11:12  Error rate at 0%
- 11:25  Incident resolved

## Root Cause Analysis
1. Why did token validation fail?
   The JWT secret rotated before existing tokens expired.
2. Why did the secret rotate?
   The config change reduced the rotation interval from 24h to 1h.
3. Why did the change pass tests?
   Config tests did not validate secret rotation behavior.
4. Why was there no alert before impact?
   The latency alert threshold was set too high.

## What went well
- Fast detection and response (3 min from detection to declaration)
- Clear communication to stakeholders and support
- Quick and effective rollback

## What went wrong
- Config change without security review
- Insufficient config tests
- Alert thresholds too high

## Action Items
| Action | Owner | Due Date | Priority |
|--------|-------|----------|----------|
| Add secret rotation test | alice | 2026-07-18 | P0 |
| Review latency alert thresholds | bob | 2026-07-15 | P1 |
| Add security review for config changes | platform | 2026-07-25 | P1 |

Variants

Context	Adjustments	Notes
Security incident	Add impact assessment section for data exposure, add legal review, and restrict distribution	Security postmortems may be confidential
Data loss incident	Add data recovery steps, backup verification, and customer notification timeline	Focus on what was lost and what was recovered
Performance degradation (not outage)	Add latency percentiles, throughput drop, and cascading slowdown effects	Degradation is harder to define than downtime
Third-party dependency failure	Add vendor communication timeline and alternative provider evaluation	You cannot fix the vendor, but you can reduce dependence
Recurring incident	Add a comparison to previous similar incidents and a deeper systemic analysis	Patterns matter more than individual events

What works

1. Schedule within 48 hours — memory fades and logs rotate; run the postmortem while details are fresh
2. Invite observers, not just responders — people not in the heat of the moment often see patterns responders miss
3. Focus on systems, not people — “the alert was missed” is a symptom; “the alert was drowned in noise” is a system problem
4. Publish action items in the same week — the value of a postmortem is proportional to how fast its actions are tracked
5. Review old postmortems quarterly — look for recurring themes and systemic gaps

Common Mistakes

1. Root cause = human error — humans are the most variable component; the system should have made the error safe
2. No executive summary — without a one-paragraph summary, leadership will not read the rest
3. Action items without owners or dates — unassigned actions are forgotten actions
4. Skipping “what went well” — postmortems are not just complaints; they reinforce practices that worked
5. No follow-up — if nobody checks whether action items are done, the postmortem was a waste of time

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the postmortem and incident guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply postmortem incident review template when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Common Production Pitfalls

• Leaving required fields blank or using vague one-word answers.
• Filling the document once and never updating it after scope or decisions change.
• Storing the document where the team does not look during incidents or reviews.
• Not assigning an owner, due date, or review cadence.
• Copying boilerplate without removing sections that do not apply.
• Skipping version control, which makes rollback and accountability impossible.
• Failing to link the document to related decisions or follow-up actions.
• Avoiding quarterly reviews that would retire stale or unused sections.