Incident Postmortem Template

Overview

Think of a postmortem as the incident’s written record: what happened, why it happened, how it got resolved and what changes prevent a repeat. Done well, it’s blameless: the analysis goes after systemic causes, not individual mistakes. With one shared structure, every incident ends up with a consistent, actionable postmortem.

When to Use

• After any P0 or P1 incident (required within 48 hours)
• After recurring P2 incidents (third occurrence triggers postmortem)
• After near-miss events that could have caused significant impact
• After security incidents or data breaches
• For compliance and audit requirements

Before You Start

• The incident is fully resolved and systems are stable.
• A timeline of events has been collected — see the incident timeline template.
• Key participants are available: responders, involved engineers, and at least one observer.
• Leadership supports a blameless process.
• It’s decided whether the postmortem is internal-only or customer-facing.

Solution

# Incident Postmortem: `<Incident ID>`

## Incident Summary

| Field | Value |
|-------|-------|
| Incident ID | INC-2026-07-05-001 |
| Title | Payment service outage during checkout peak |
| Severity | P0 |
| Status | Resolved |
| Date | 2026-07-05 |
| Start Time | 14:32 UTC |
| Detection Time | 14:35 UTC (3 min to detect) |
| Mitigation Time | 14:52 UTC (20 min to mitigate) |
| Resolution Time | 15:48 UTC (76 min total) |
| Incident Commander | <Name> |
| Postmortem Author | <Name> |
| Postmortem Reviewers | <Tech Lead>, <SRE Lead> |
| Affected Services | Payment Service, Checkout Service |
| Affected Users | ~12,000 active users |
| Revenue Impact | ~$45,000 lost orders |

## Impact Summary

### User-Facing Impact

| Impact | Duration | Affected Users |
|--------|----------|----------------|
| Checkout page returned 500 error | 20 minutes (14:32-14:52) | ~12,000 |
| Payment processing failed | 20 minutes | ~3,500 attempted checkouts |
| Order confirmation emails delayed | 76 minutes | ~1,200 orders queued |
| Refund processing delayed | 76 minutes | ~50 pending refunds |

### Business Impact

| Metric | Normal | During Incident | Impact |
|--------|--------|-----------------|--------|
| Orders per minute | 180 | 0 (for 20 min) | 3,600 lost orders |
| Revenue per minute | $2,250 | $0 (for 20 min) | $45,000 lost revenue |
| Support tickets | 5/hour | 85/hour | 80 extra tickets |
| Customer complaints | 2/hour | 45/hour | 43 extra complaints |
| NPS impact | — | -8 points (estimated) | Survey in 1 week |

### SLO Impact

| SLO | Target | Current (30d) | Before Incident | After Incident | Budget Consumed |
|-----|--------|---------------|-----------------|----------------|-----------------|
| Availability | 99.9% | 99.85% | 99.92% | 99.85% | 70% of monthly budget |
| Latency p95 | < 500ms | 680ms | 420ms | 680ms | N/A (separate SLO) |

## Timeline

| Time (UTC) | Event | Source |
|------------|-------|--------|
| 14:30 | Deployment v2.5.0 goes live — includes new payment provider integration | CI/CD pipeline |
| 14:32 | Error rate spikes from 0.1% to 45% on `/api/v1/payments` endpoint | Prometheus alert |
| 14:35 | PagerDuty alert fires: `HighErrorRate-PaymentService` | Alertmanager |
| 14:36 | On-call engineer acknowledges alert | PagerDuty |
| 14:38 | On-call engineer checks Grafana dashboard — sees 500 errors on payment endpoint | Grafana |
| 14:40 | On-call engineer checks logs — sees `TypeError: Cannot read property 'id' of undefined` in payment service | Kibana |
| 14:42 | On-call engineer identifies new deployment as likely cause — checks `#deployments` channel | Slack |
| 14:44 | On-call engineer confirms v2.5.0 deployed at 14:30, 2 minutes before errors started | CI/CD logs |
| 14:46 | On-call engineer escalates to secondary on-call for rollback support | PagerDuty |
| 14:48 | Decision made to roll back to v2.4.1 | Incident channel |
| 14:50 | Rollback initiated: `kubectl rollout undo deployment/payment` | Terminal |
| 14:52 | Rollback complete — error rate drops to 0.1% | Grafana |
| 14:55 | Error rate confirmed normal for 3 consecutive minutes | Grafana |
| 15:00 | Incident downgraded from P0 to P1 — monitoring for residual issues | Incident channel |
| 15:15 | Order confirmation email queue processing begins — 1,200 emails queued | Email service |
| 15:30 | Email queue cleared — all confirmations sent | Email service |
| 15:48 | All pending refunds processed — incident fully resolved | Payment service |
| 16:00 | Incident declared resolved in PagerDuty | PagerDuty |
| 16:30 | Postmortem scheduled for 2026-07-06 10:00 UTC | Calendar |

## Root Cause Analysis

### What Happened

Deployment v2.5.0 introduced a new payment provider integration. The new code expected the payment provider's API to always return a `transaction.id` field in the response. However, for certain card types (Amex, Discover), the provider returns the transaction ID in a nested object: `transaction.metadata.transaction_id`. The code accessed `transaction.id` directly, which was `undefined` for these card types, causing a `TypeError` that crashed the request handler.

### Why It Happened

| Factor | Detail |
|--------|--------|
| Insufficient test coverage | Integration tests used mocked responses that always included `transaction.id`. No test covered the nested response format. |
| No contract testing | The payment provider's API documentation was not verified against actual responses. No contract test validated the response schema. |
| Missing defensive coding | The code did not validate the response structure before accessing nested fields. No null check on `transaction.id`. |
| Inadequate canary deployment | The deployment went to 100% of traffic immediately. No canary or gradual rollout was configured. |
| No feature flag | The new payment provider integration was not behind a feature flag. It could not be disabled without a rollback. |

### Five Whys Analysis

1. **Why did the payment service return 500 errors?**
   The code threw a `TypeError` when accessing `transaction.id` on undefined.

2. **Why was `transaction.id` undefined?**
   The payment provider returns transaction IDs in `transaction.metadata.transaction_id` for Amex and Discover cards, not in `transaction.id`.

3. **Why didn't tests catch this?**
   Integration tests used mocked responses that always included `transaction.id`. No test used the actual response format for non-Visa/Mastercard types.

4. **Why wasn't the actual response format verified?**
   No contract testing was set up between the payment service and the payment provider. The team relied on API documentation, which was outdated.

5. **Why did the deployment affect all users immediately?**
   No canary deployment or gradual rollout was configured. The deployment went to 100% of traffic in one step.

### Contributing Factors

| Factor | Impact | Category |
|--------|--------|----------|
| Deployment during peak hours (14:30 UTC) | Maximized user impact | Process |
| No deployment freeze during peak window | Allowed risky deploy during high traffic | Policy |
| Alert threshold too high (5% error rate) | 3-minute detection delay | Monitoring |
| No synthetic check for payment flow | No proactive detection before users hit errors | Monitoring |
| Email queue backed up | Delayed order confirmations for 76 minutes | Architecture |
| Single payment provider | No fallback when integration broke | Architecture |

## What Went Well

- [x] Alert fired within 3 minutes of error start
- [x] On-call engineer acknowledged within 1 minute
- [x] Root cause identified within 10 minutes
- [x] Rollback decision made within 16 minutes
- [x] Rollback completed within 20 minutes
- [x] Communication in incident channel was clear and timely
- [x] No data corruption — all orders were preserved in the database

## What Went Wrong

- [x] Deployment went to 100% traffic without canary
- [x] Tests did not cover non-Visa/Mastercard response formats
- [x] No contract testing with payment provider
- [x] No feature flag to disable new integration without rollback
- [x] Email queue had no backpressure mechanism — 1,200 emails delayed
- [x] Alert threshold at 5% meant 3 minutes of errors before alerting

## Where We Got Lucky

- [x] The rollback was fast because the previous version was still available
- [x] The error was consistent and easy to reproduce — diagnosis was straightforward
- [x] No data was lost — orders were saved to the database before the crash
- [x] The payment provider was not down — only our integration was broken

## Action Items

| # | Action | Type | Owner | Priority | Due Date | Status |
|---|--------|------|-------|----------|----------|--------|
| 1 | Add contract tests for payment provider API responses | Prevent | Backend team | High | 2026-07-12 | Open |
| 2 | Add test cases for Amex, Discover, and other card types | Prevent | Backend team | High | 2026-07-10 | Open |
| 3 | Implement canary deployment for payment service | Prevent | Platform team | High | 2026-07-19 | Open |
| 4 | Add feature flag for new payment provider integration | Prevent | Backend team | Medium | 2026-07-15 | Open |
| 5 | Lower error rate alert threshold from 5% to 2% | Improve | SRE team | Medium | 2026-07-08 | Open |
| 6 | Add synthetic check for payment flow (every 1 min) | Improve | SRE team | Medium | 2026-07-12 | Open |
| 7 | Implement backpressure in email queue | Improve | Platform team | Low | 2026-07-26 | Open |
| 8 | Add deployment freeze policy for peak hours (9-18 UTC) | Process | Eng Manager | Medium | 2026-07-10 | Open |
| 9 | Update runbook with canary deployment procedure | Process | SRE team | Low | 2026-07-19 | Open |
| 10 | Evaluate secondary payment provider as fallback | Architect | Backend team | Low | 2026-08-05 | Open |

### Action Item Categories

| Category | Description | Count |
|----------|-------------|-------|
| Prevent | Prevent this specific issue from recurring | 4 |
| Improve | Improve detection, response, or mitigation | 3 |
| Process | Improve processes or policies | 2 |
| Architect | Architectural changes to reduce blast radius | 1 |

## Lessons Learned

### Technical Lessons

1. **Mock responses must match real API behavior** — Mocked tests gave false confidence. Contract tests against the real API would have caught the schema mismatch.
2. **Canary deployments catch integration bugs** — A 5% canary would have affected ~600 users instead of 12,000.
3. **Feature flags enable instant rollback** — A feature flag would have disabled the new integration in seconds, no deployment rollback needed.
4. **Synthetic checks detect issues before users** — A synthetic check hitting the payment flow every minute would have detected the error before real users.

### Process Lessons

1. **Deployment timing matters** — Deploying during peak hours maximizes impact. A deployment freeze during peak hours is a simple policy change.
2. **Alert thresholds should be tuned** — A 5% error rate threshold is too high for a critical service. 2% would have alerted 2 minutes earlier.
3. **Backpressure prevents cascading failures** — The email queue backed up because there was no backpressure mechanism. This extended the incident beyond the service recovery.

## Appendix

### Error Logs (Excerpt)

```
2026-07-05T14:32:01.234Z ERROR [payment-service] TypeError: Cannot read properties of undefined (reading 'id')
    at processPaymentResponse (payment.js:142)
    at handlePayment (payment.js:87)
    at async processCheckout (checkout.js:34)
    at async <anonymous> (server.js:156)
  transactionId: undefined
  cardType: "amex"
  requestId: "req-abc123"
```

### Alert Configuration

```yaml
- alert: HighErrorRate-PaymentService
  expr: |
    rate(http_requests_total{service="payment",status=~"5.."}[5m])
    /
    rate(http_requests_total{service="payment"}[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
```

### Deployment Details

| Field | Value |
|-------|-------|
| Deployed Version | v2.5.0 |
| Previous Version | v2.4.1 |
| Deploy Method | kubectl apply (100% rollout) |
| Deploy Time | 14:30 UTC |
| Changes | New payment provider integration (Stripe → Adyen) |
| PR | #1234 |
| Reviewed By | 2 reviewers |
| CI Status | All checks passed |

Explanation

A blameless postmortem goes after systemic causes, not individual actions. Nobody ships code meaning to break production; the system let a risky change reach every user without adequate testing. The postmortem’s job is to name the systemic changes that would have stopped it.

The timeline is the factual record, built from objective sources only: PagerDuty logs, CI/CD records, chat messages, monitoring data. Subjective interpretation doesn’t belong there.

The five whys digs past the symptoms. Keep asking until you land on a systemic cause that, fixed, would have prevented the incident entirely.

Action items are the part that matters most. Give each one an owner, a priority and a due date. Before writing a new postmortem, review the open items from previous ones — recurring items are the telltale sign of a follow-through problem.

Second Worked Example: Config Change Without a Deploy

Not every incident starts with new code. Condensed postmortem for a JWT secret rotation change:

• What happened: 15% of EU login attempts failed for 30 minutes after a config change shortened the JWT secret rotation interval from 24h to 1h, invalidating active sessions.
• Timeline: DB CPU rose 10:42, PagerDuty fired 10:55, SEV-1 declared 11:00, config change identified 11:03, rollback complete 11:08, resolved 11:25 UTC.
• Five whys: token validation failed → the secret rotated before existing tokens expired → the config change passed tests → config tests never covered rotation behavior → no alert fired before user impact → latency thresholds were set too high. The root cause is the missing review process for config changes, not the change itself.
• Action items: add a secret-rotation test (P0), retune latency alert thresholds (P1), require security review on config changes (P1).

Same shape as the payment example — trigger, chain of whys, systemic fix — but the trigger is operational, not a code bug.

Running the Postmortem Meeting

Assign a facilitator who wasn’t an incident responder — they bring a fresh perspective and can push past “we all know what happened.” Start with the timeline so the group agrees on facts before interpreting them, then work the five whys out loud. Encourage questions, not accusations, cap the session at 60 minutes, and document in a shared file in real time so nothing is reconstructed from memory. Assign action items with owners and due dates before the meeting ends, and share the draft with all engineering teams within 48 hours.

Invite observers as well as responders: people who were not in the heat of the incident routinely spot patterns that responders missed.

Variants

Context	Approach	Notes
Security incident	Add CVE, attack vector, data exposure details	Follow security IR plan
Data loss incident	Add data recovery steps, backup verification	Include data integrity checks
Third-party outage	Add vendor communication timeline	Include vendor status page timeline
Near-miss	Skip impact section, focus on prevention	No user impact to document
Multi-service incident	Add service dependency map	Show cascade path
Performance degradation (not outage)	Add latency percentiles, throughput drop, and cascading slowdown effects	Degradation is harder to bound than downtime
Recurring incident	Compare to previous postmortems and analyze why prior actions didn’t prevent it	Patterns matter more than individual events

What Works

1. Write postmortems within 48 hours; details fade fast.
2. Keep them blameless: blame the system, never the person.
3. Build the timeline from objective sources (PagerDuty, CI/CD, monitoring).
4. Assign every action item an owner and a due date, since unowned items don’t get done.
5. Review previous action items first; recurring items signal a follow-through problem.
6. Share postmortems broadly so other teams learn from your incidents.
7. Track the completion rate; a postmortem without completed actions is theater.
8. Publish action items in the same week — a postmortem’s value decays with the delay.
9. Review old postmortems quarterly to catch recurring themes and systemic gaps.

Common Mistakes

1. Blaming individuals: “the engineer should have tested more” isn’t a root cause.
2. Vague action items like “improve testing” go nowhere without specifics and owners.
3. No follow-up, which turns the whole postmortem into wasted effort.
4. Writing it too late, once memories have faded and the timeline turns fuzzy.
5. Only fixing prevention, when better detection and mitigation count just as much.
6. Skipping “what went well,” the part that reinforces the responses you want repeated.
7. Keeping postmortems private instead of letting other teams learn from them.

See Also

• Companion repository: the fillable template and the filled payment-outage example as standalone markdown files.
• Blameless Postmortems: Learning from Incidents Without Blame: the companion guide on how the postmortem process actually runs.
• Google SRE — Postmortem Culture: the original blameless postmortem chapter from the SRE book.
• Atlassian Incident Handbook: incident and postmortem playbooks used at real scale.