Disaster Recovery Plan Template

Use this template to prepare for catastrophic failures and minimize recovery time. Complement it with the Runbook Template for operational procedures, and verify your backups actually restore with the Backup Verification Test Template.

Template

# Disaster Recovery Plan: [Service / System Name]

## Overview
| Field | Value |
|-------|-------|
| **Plan owner** | [team or individual] |
| **Last tested** | [date] |
| **RTO** | [hours — max acceptable downtime] |
| **RPO** | [minutes — max acceptable data loss] |
| **Declaration authority** | [who can declare a disaster] |

## Risk Scenarios

| Scenario | Likelihood | Impact | Mitigation |
|----------|-----------|--------|------------|
| Region outage | Medium | Critical | Multi-region deployment |
| Database corruption | Low | Critical | Point-in-time restore |
| Credential compromise | Medium | High | Token rotation + IAM lockdown |
| Third-party outage | High | Medium | Circuit breakers + fallback |

## Failover Procedures

### Scenario: Primary Region Unavailable

1. **Detect** — monitoring alert confirms regional health check failure
2. **Decide** — incident commander confirms failover (not flapping)
3. **Route** — update DNS / load balancer to secondary region
4. **Verify** — smoke tests pass in secondary region
5. **Communicate** — update status page and internal channels

### Scenario: Database Corruption

1. **Stop writes** — set database to read-only
2. **Identify** — determine corruption scope and time of event
3. **Restore** — restore from latest clean backup to new instance
4. **Replay** — apply WAL / binlog up to just before corruption
5. **Validate** — run data integrity checks
6. **Switch** — promote restored instance to primary

## Recovery Runbook

​```bash
# 1. Verify secondary region health
kubectl --context=dr get nodes

# 2. Promote read replicas
gcloud sql instances promote-replica dr-replica

# 3. Update DNS
aws route53 change-resource-record-sets --hosted-zone-id Z123 \
  --change-batch file://failover.json

# 4. Verify
./smoke-tests.sh --env=dr
​```

## Dependencies and Their DR Status

| Dependency | Their RTO | Our Fallback |
|------------|-----------|-------------|
| Payment processor | 4 hours | Queue transactions, retry later |
| Identity provider | 1 hour | Cached JWT validation + degraded login |
| CDN | 0 minutes | Multi-CDN switch |

## Roles During a Disaster

| Role | Responsibility |
|------|---------------|
| Incident commander | Declares the disaster, approves failover |
| Recovery lead | Executes the runbook steps |
| Communications lead | Status page, stakeholders, customers |
| Scribe | Timelines every action for the postmortem |

## Testing Schedule

| Test Type | Frequency | Last Completed | Result |
|-----------|-----------|----------------|--------|
| Tabletop exercise | Quarterly | [date] | [pass / gaps found] |
| Failover drill | Bi-annually | [date] | [pass / gaps found] |
| Backup restore test | Monthly | [date] | [pass / gaps found] |
| Chaos engineering | Monthly | [date] | [pass / gaps found] |

## Communication Plan

| Audience | Trigger | Message | Channel |
|----------|---------|---------|---------|
| Engineering | Any DR event | Incident channel | Slack #incidents |
| Leadership | RTO > 50% | Status update | Email + call |
| Customers | RTO > 75% | Status page + tweet | Status page |

RTO and RPO Explained

Metric	Definition	Example
RTO (Recovery Time Objective)	How long you can be down	4 hours
RPO (Recovery Point Objective)	How much data you can lose	15 minutes

These two numbers drive every other decision in the plan — and they’re where most DR plans go wrong. Teams copy “RTO 1 hour, RPO 15 minutes” from another document without checking whether their infrastructure can physically deliver it.

The math is unforgiving: if your database backups run every hour and your RPO is 15 minutes, your backup strategy doesn’t meet the objective — you need continuous replication or WAL archiving. If your DNS TTL is 300 seconds and failover requires a DNS change, that alone eats 5 minutes of a 15-minute RTO before anyone touches a runbook.

Set targets per service tier, not globally:

Tier	Example services	Suggested RTO	Suggested RPO
Tier 0 — revenue-critical	checkout, auth, payments	5-15 min	~0 (sync replication)
Tier 1 — user-facing	API, web app, search	30-60 min	5-15 min
Tier 2 — internal tools	dashboards, admin, analytics	4-8 hours	1-24 hours
Tier 3 — batch/reporting	ETL jobs, exports	24-48 hours	24-48 hours

Getting buy-in for these numbers matters more than the numbers themselves. An RTO set by management without engineering input tends to be physically impossible; an RTO set by engineering without business input tends to over-invest. The SLO Document Template helps formalize the uptime targets these DR objectives should support.

Choosing a DR Architecture

Four standard patterns, ordered by cost:

Pattern	How it works	RTO achievable	Relative cost
Backup & restore	Snapshots to another region; rebuild on failure	Hours-days	$
Pilot light	Minimal core (DB replication) always on; scale up on failover	30-60 min	$$
Warm standby	Scaled-down live copy in secondary region	Minutes	$$$
Active-active	Full capacity in both regions, traffic split	Near-zero	$$$$

The trap is paying for warm standby everywhere. Match the architecture to the tier table above: active-active for Tier 0, pilot light for most Tier 1 services, backup-and-restore for everything else. A startup that skips Tier 3 analysis ends up running production databases for services that would be fine restored from a nightly snapshot.

Two details that kill otherwise good architectures: data replication lag (async replication means your warm standby can be minutes behind — that’s your real RPO) and split-brain (if both regions accept writes during a partition, merging them is a manual nightmare; pick one primary and fence the other).

The Backup Layer

Every DR plan sits on top of a backup strategy, and most teams have one that’s weaker than they think. The classic rule is 3-2-1: 3 copies of your data, on 2 different media (or providers), with 1 copy offsite (or offline). A database snapshot in the same region as the database is two copies in one location — it protects against corruption, not against a region outage or an account compromise.

Three details worth auditing before you write the plan:

• Immutability. Ransomware deletes backups first. Snapshots that can be deleted from the same credentials that administer production aren’t a recovery path — they’re part of the blast radius. Object-lock or a separate account for backup storage closes that hole.
• Restore time is the real RPO+RTO input. A 2TB snapshot that takes 6 hours to restore makes a 1-hour RTO physically impossible no matter what the plan says. Measure restore duration during drills, not during the incident.
• Point-in-time recovery beats snapshots for corruption scenarios. If someone dropped a table at 09:00 and you notice at 14:00, the last snapshot contains the mistake. WAL/binlog archiving lets you replay up to just before it.

Declaring a Disaster

The hardest DR decision isn’t technical — it’s deciding that today is the day. Trigger too early and you fail over on a false positive, possibly making things worse; too late and the RTO is gone before anyone opened the runbook.

The template’s “declaration authority” field exists because this decision can’t be a committee vote mid-incident. Write down in advance who can declare (usually the incident commander or the on-call lead, never a group), and give them criteria they can check in under five minutes: a confirmed regional health check failure, a data integrity check that fails, or a vendor confirming a multi-hour outage on a critical dependency.

Equally important: define what does not count as a disaster. A degraded region where your app still serves reads is an incident, not a DR event. Failing over on a degraded-but-alive primary is how teams turn a 20-minute cloud hiccup into a two-hour self-inflicted outage.

Running a Tabletop Exercise

A tabletop exercise is a paper drill: gather the on-call team for 60-90 minutes and walk the plan aloud, step by step, pretending a real scenario just fired. It costs almost nothing and finds 80% of the problems a live drill would.

Structure it like this:

1. Pick one scenario from your risk table — “primary region unavailable” is the classic first run.
2. Narrate the timeline. “It’s 14:03, the alert fires. Who sees it? What do they do first?” Force concrete answers, not “we’d figure it out.”
3. Assign a scribe to write down every gap: a phone number that’s stale, a DNS credential only one person has, a step nobody knew existed.
4. End with an action list — owner and deadline for each gap. A tabletop without follow-ups is a meeting, not a test.

Run the drill with someone who didn’t write the plan. If only the author can execute it, the plan isn’t a plan — it’s one person’s memory.

Keeping the Plan Alive

A DR plan decays faster than almost any other document on the wiki. Contacts leave, regions get renamed, the Slack channel gets archived, someone rotates the DNS credentials — and the plan keeps pointing at a world that no longer exists. The only defense is a review cadence that runs on a calendar, not on someone’s memory.

Put the review inside the drill schedule, not next to it: every tabletop ends with ten minutes of “what in this document is now wrong.” That catches staleness while it’s cheap. Twice a year, verify the small stuff explicitly — phone numbers, on-call rotations, account IDs, DNS credentials — because those are exactly the details that silently rot between big incidents.

Version the plan in git like code. When a real incident happens, you want to diff what the plan said against what the team actually did — that gap is the most honest input the next revision will ever get.

And assign a single owner to the document. Not a committee — one named person whose job includes pinging the review when the calendar says so. Shared ownership of a DR plan reads on paper like collaboration and behaves in practice like abandonment. If that person leaves and nobody inherits the document, you’ll find out the plan was orphaned exactly when you needed it most.

What Works

• Test recovery quarterly — an untested plan is a fantasy. See Infrastructure as Code Guide for automated environment provisioning.
• Automate failover where possible — human-driven failover takes 10x longer under stress.
• Document decisions, not just steps — why you chose this RTO helps future reviewers push back correctly.
• Keep the plan accessible offline — during a disaster, your internal wiki may be down. Reference Monitoring and Alerting Guide for detection triggers.
• Include third-party dependencies — your DR is only as strong as your weakest vendor.
• Time the drill against the target — if your RTO is 1 hour and the drill takes 3, that’s a finding, not a failure. Log the gap, fix the bottleneck, and run it again; a plan that only meets its targets on paper meets them nowhere.

Common Mistakes

• Untested backups — a backup you’ve never restored is Schrödinger’s backup.
• Single region everything — AWS regions fail; multi-region isn’t optional for critical services.
• No rollback from failover — failing back to primary is harder than failing over, and almost nobody plans it.
• Ignoring data consistency during failover — split-brain writes corrupt data on both sides.
• RTO/RPO set by managers without engineering input — if the target is physically impossible, the plan is theater.
• Copying the template once and never updating it — a DR plan with stale contact lists is worse than no plan, because it creates false confidence.
• Leaving required fields blank — a table full of [date] placeholders means the drill never actually ran.

Troubleshooting DR Procedures

• Failover runs but traffic doesn’t switch: check DNS TTL — a 300s TTL means up to 5 minutes of stale records. Lower TTLs before the drill, or use a load balancer layer that doesn’t depend on DNS propagation.
• Backup exists but won’t restore: the restore path was never exercised. Corrupted snapshots, missing dependencies, or a database version that changed since the backup. Monthly restore tests catch this before it matters.
• Drill blows past RTO: time each step separately. Usually the bottleneck isn’t the infrastructure — it’s the approval chain or a runbook step that assumes knowledge only one person has.
• Split-brain after partial failover: fence the old primary before promoting the replica (block writes at the network or IAM level), then reconcile. Never let both sides accept writes.
• Failback to primary goes wrong: nobody tested it. Plan failback as its own scenario — it’s usually harder than the failover because the “new” primary now has live data the old one lacks.

Variants

Context	Approach	Notes
Startup	Simplified DR: backup + restore	RTO 8h, RPO 24h acceptable
Enterprise	Multi-region active-active DR	RTO 15min, RPO 0
E-commerce	DR with priority on catalog and checkout	Restore in revenue-impact order
Fintech	DR with priority on transaction integrity	Zero data loss tolerable

DR Plan Example: Primary Region Failure

=== DR Plan: us-east-1 Region Failure ===

Scenario: us-east-1 region unavailable
RTO Target: 1 hour
RPO Target: 15 minutes

Initial Triage (0-5 min):
  1. Confirm us-east-1 is down (not a false positive)
     - Check AWS Health Dashboard
     - Check status.aws.amazon.com
  2. Declare DR-SEV1 incident
  3. Notify leadership and stakeholders
  4. Activate DR channel (#dr-incident)

Failover Execution (5-45 min):
  Step 1: Promote secondary region (eu-west-1) to primary
    - Update DNS (Route 53) to point to eu-west-1
    - Verify health checks pass in eu-west-1
    - Estimated time: 5-10 min

  Step 2: Scale resources in eu-west-1
    - Increase replicas to handle full traffic
    - kubectl scale deployment app -n production --replicas=20
    - Estimated time: 5-10 min

  Step 3: Restore data from backup
    - Last RDS backup: 12 min ago (within RPO)
    - Restore snapshot to new instance in eu-west-1
    - Estimated time: 15-20 min

  Step 4: Verify services
    - Health checks on all endpoints
    - Smoke tests on critical flows (login, payment, checkout)
    - Verify no data loss (compare record counts)
    - Estimated time: 5-10 min

Communication (parallel to execution):
  - Status page: "Investigating us-east-1 outage"
  - At 15 min: "Failover to eu-west-1 in progress"
  - At 45 min: "Services restored in eu-west-1"
  - At 60 min: "Incident resolved; investigating root cause"

Rollback (if failover fails):
  - If eu-west-1 cannot handle load: degrade to read-only mode
  - If data cannot be restored: use last valid backup (higher RPO)
  - If all fails: activate prolonged outage communication plan

Post-Recovery:
  - Monitor stability in eu-west-1 for 24 hours
  - Investigate root cause of us-east-1 failure
  - Conduct postmortem within 48 hours
  - Update DR plan with learnings
  - Plan migration back to us-east-1 (or new region)

Key Takeaways

• An untested DR plan is fiction — the quarterly tabletop is what turns a document into a capability.
• RTO/RPO are engineering constraints, not aspirations: check your backup cadence and DNS TTLs against them before writing them down.
• Match the DR architecture to the service tier — active-active everywhere is budget theater.
• Human approval for the traffic switch, automation for everything around it.
• Failback deserves its own plan; most teams discover this during their first real incident.

Further Reading

• AWS — Disaster Recovery of Workloads on AWS — the reference whitepaper for backup/restore, pilot light, warm standby, and active-active.
• Google SRE Workbook — Incident Response — how Google’s SRE teams structure incident response and postmortems.
• Cloudflare and GitLab postmortem archives — real DR events and what actually went wrong.