Cross-Region Failover Test Template

Take this template into your next cross-region failover drill and leave with evidence instead of assumptions. It sits one level below the Disaster Recovery Test Plan, which defines program scope and compliance cadence; this page is the execution document for moving one service from its primary region to a secondary and back.

Overview

A failover test that ends with “traffic moved, looks fine” tells you almost nothing. A useful test leaves a record: how long each stage took, where the plan diverged from reality, and whether you managed to get back. This template forces that discipline: it assigns roles, sets go/no-go gates before anyone touches production, times each step against the RTO target, and treats failback as part of the test rather than an afterthought.

A few terms worth pinning down before you copy anything:

• Failover is moving traffic and write authority from the primary region to the secondary. In a drill it’s planned; a real incident never is, which is the point of drilling.
• Failback is the return trip: restoring replication, re-promoting the original primary, and shifting traffic back without losing the data written while the secondary was in charge.
• RTO (recovery time objective) is how long the service can be degraded or down. Measure it from the moment the decision clock starts, not from the first command you run.
• RPO (recovery point objective) is how much data you can afford to lose, driven directly by replication lag at the moment of promotion.
• Split-brain is the failure mode where both regions accept writes at once. Preventing it comes down to fencing, not DNS.

The template assumes an AWS-shaped stack (Route 53, Aurora, Auto Scaling) because that’s what the script targets, but every section maps to an equivalent on GCP or Azure. Where it doesn’t apply to your stack, delete the section rather than filling it with text nobody will read during an incident.

When to Use

Use this template when:

• A service has a contractual or regulatory RTO/RPO and you need a documented drill to prove you can meet it. Auditors ask for evidence, and a completed copy of this document is evidence.
• You’re bringing a secondary region online and want to validate it under controlled conditions before it matters.
• Infrastructure changed since the last test: new database engine, new DNS provider, a migrated load balancer, or a team that’s never run the procedure.
• A past incident or a game day exposed gaps, and you want proof they’re closed.

Skip it when:

• The failure you’re worried about is a single availability zone. That’s what multi-AZ handles; the scenario deserves its own, cheaper drill.
• The service is genuinely stateless and already runs active-active across regions. Nothing moves in that setup; you’re trusting health checks, not a procedure.
• Cross-region replication isn’t set up on the data layer yet. Fix that first; a failover drill against a cold secondary measures provisioning speed, not recovery.
• The bridge has to be staffed for the whole window; a half-attended drill produces half-trusted results.

The Template

Copy the block below into your runbook, wiki, or incident tool and fill in every bracketed value before the test window opens. Anything left blank becomes a decision someone makes under pressure, which is exactly what the template is meant to prevent.

# Cross-Region Failover Test: `<service-name>`

## 1. Service Metadata

| Field | Value |
|-------|-------|
| Service | `<service-name>` |
| Primary region | `us-east-1` |
| Secondary region | `eu-west-1` |
| RTO target | `15 min` |
| RPO target | `5 min` |
| Data replication | `Aurora Global / async` |
| Last test date | `YYYY-MM-DD` |
| Test owner | `@team-or-person` |

## 2. Roles

| Role | Person | Responsibility |
|------|--------|----------------|
| Test lead | | Owns the clock, calls go/no-go, can abort |
| Executor | | Runs the script, reports each step |
| Data validator | | Checks replication lag and integrity |
| Comms | | Status page, stakeholder channel |
| Observer | | Logs timeline, notes surprises |

## 3. Go/No-Go Criteria

Test proceeds only if ALL are true at T-30 min:

- [ ] Secondary region healthy (compute, database replica, endpoints)
- [ ] Replication lag < 50% of RPO target
- [ ] DNS record TTL ≤ 60s (checked with `dig`, not assumed)
- [ ] Failback procedure reviewed and executor knows it
- [ ] Stakeholders notified, status page ready
- [ ] Someone has authority to abort and will say so

## 4. Abort Criteria

Abort immediately and fail back (or hold state) if:

- Real production incident starts in either region
- Replication lag exceeds RPO after promotion decision
- Data integrity check fails post-failover
- Any step exceeds its time budget by 2x without a known cause

## 5. Failover Timeline

| T+ | Action | Verify | Budget |
|----|--------|--------|--------|
| 0 | Confirm failure signal / declare drill start | Two independent signals | 2 min |
| 2 | Fence primary: block writes, stop writers | No write connections | 3 min |
| 5 | Promote secondary database | Accepts writes | 5 min |
| 10 | Scale secondary compute to prod size | Desired capacity reached | 5 min |
| 15 | Update DNS / routing to secondary | `dig` from 2+ locations | 2 min |
| 17 | Synthetic transactions against service | 100% success, 5 min | 5 min |
| 22 | Declare failover complete, start soak | — | — |
| 52 | End soak, start failback | See section 6 | — |

## 6. Failback

| Step | Action | Verify |
|------|--------|--------|
| 1 | Re-establish replication secondary → primary | Lag trending to 0 |
| 2 | Wait until lag = 0, verify checksums | Row counts match |
| 3 | Fence secondary writers | No write connections |
| 4 | Switch DNS back to primary | `dig` confirms |
| 5 | Scale secondary back to standby | Desired capacity |
| 6 | Confirm steady-state replication | Lag < threshold |

## 7. Post-Test Report

| Metric | Target | Actual | Pass |
|--------|--------|--------|------|
| Detection to decision | < 2 min | | |
| Decision to promoted | < 10 min | | |
| Total failover time | < RTO | | |
| Data loss (lag at promotion) | < RPO | | |
| Failback time | < 30 min | | |
| Errors during cutover | < 0.5% | | |

### Surprises and deltas

- What took longer than budgeted and why:
- Which step had no written procedure:
- Which alarm did or didn't fire:
- Action items (owner, due date):

How the Test Fits Together

Each section of the template exists because a real failover once failed there. The roles table exists because an unpaged executor turns a 15-minute RTO into a 40-minute search for the right person. The go/no-go gate exists because drills started on a sick secondary produce a failed test and a confused report. The abort criteria exist because someone has to be able to stop the exercise when real production trouble starts mid-test. The three parts below cover where the mechanics and the clock collide hardest in a failover.

RTO and RPO in practice

Think of RTO as a budget you’re spending. If your target is 15 minutes and detection alone consumes 5, everything downstream gets 10 minutes total. That’s why the timeline table budgets each step: the first test almost always reveals that the bottleneck isn’t the database promotion you worried about, it’s the two people who weren’t sure who was allowed to press go. Write the decision step into the timeline explicitly, with a name attached.

RPO is set by your replication technology, not by what you write in the document. Aurora Global Database reports lag through the AuroraGlobalDBRPOLag CloudWatch metric in milliseconds; PostgreSQL streaming replication exposes replay_lag on the standby. Check the actual metric before the test, not a nominal value from the architecture diagram. If the number on test day is worse than your stated RPO, the honest outcome is “no-go” or a documented exception, not a quiet override.

For a planned drill on Aurora Global Database, prefer switchover-global-cluster, which synchronizes the secondary before promotion and gets you RPO = 0. Reserve failover-global-cluster for the real disaster path, since it accepts data loss up to the current lag. Testing the switchover path exercises nearly every dependency that matters and keeps the drill recoverable.

DNS is usually the bottleneck

Compute and databases can be promoted in minutes; DNS answers to a clock you set months ago. If the failover record’s TTL is 3600 seconds, a resolver that cached the old answer at T-1 keeps sending users to a dead region for up to an hour. Route 53 health checks with failover routing policies remove the manual step entirely, and latency-based or geolocation routing spreads the blast radius for a global audience.

Two practices pay for themselves: keep the failover record’s TTL at 60 seconds or less year-round (the cost is more DNS queries, which is negligible), and during the test, measure actual propagation from several vantage points with dig against public resolvers like 8.8.8.8 and 1.1.1.1. The number you record becomes next year’s real-world propagation estimate.

Fencing before promotion

Split-brain is the worst outcome a drill can produce: both regions writing at once, with divergent data to reconcile afterwards. The sequence matters more here than any single command. Block writes at the primary first (security group rule, database parameter, connection string revocation, whatever your stack supports), confirm write connections drain to zero, and only then promote the secondary. If the primary is truly unreachable, document that fencing was assumed rather than verified, and flag the residual risk in the report.

Soak before failback

Hold off on failing back right away. A 30-minute soak on the secondary surfaces the problems that only appear under real traffic: cold caches, missing secrets, a queue consumer that was never configured in region B. Those findings are the point of the drill.

Failover Execution Script

The script below automates the mechanical steps so the humans can watch and decide. Every value comes from an environment variable, --dry-run prints each action without touching anything, and a live run refuses to proceed until someone types FAILOVER at the prompt.

#!/usr/bin/env bash
# cross-region-failover.sh — controlled failover drill for Aurora Global + Route 53
# Prereqs: aws cli v2, jq, curl, dig, GNU date (gdate on macOS).
# Usage: ./failover.sh --dry-run   then   ./failover.sh
set -euo pipefail

PRIMARY_REGION="${PRIMARY_REGION:-us-east-1}"
SECONDARY_REGION="${SECONDARY_REGION:-eu-west-1}"
GLOBAL_CLUSTER_ID="${GLOBAL_CLUSTER_ID:?Set GLOBAL_CLUSTER_ID to the Aurora global cluster name}"
SECONDARY_CLUSTER_ARN="${SECONDARY_CLUSTER_ARN:?Set SECONDARY_CLUSTER_ARN for the cluster being promoted}"
SECONDARY_CLUSTER_ID="${SECONDARY_CLUSTER_ID:?Set SECONDARY_CLUSTER_ID (cluster resource name for lag metrics)}"
HOSTED_ZONE_ID="${HOSTED_ZONE_ID:?Set HOSTED_ZONE_ID for the Route 53 zone}"
DNS_RECORD="${DNS_RECORD:?Set DNS_RECORD, e.g. api.example.com}"
SECONDARY_CNAME="${SECONDARY_CNAME:?Set SECONDARY_CNAME, the regional endpoint to point at}"
SECONDARY_HEALTH_URL="${SECONDARY_HEALTH_URL:?Set SECONDARY_HEALTH_URL, e.g. https://eu-west-1.api.internal/health}"
ASG_NAME="${ASG_NAME:?Set ASG_NAME for the secondary Auto Scaling group}"
ASG_DESIRED="${ASG_DESIRED:-10}"
RPO_MAX_MS="${RPO_MAX_MS:-300000}"   # 5 minutes in milliseconds
DRY_RUN=0

for arg in "$@"; do
  case "$arg" in
    --dry-run) DRY_RUN=1 ;;
    *) echo "Unknown flag: $arg" >&2; exit 2 ;;
  esac
done

log() { printf '%s %s\n' "$(date -u +%H:%M:%SZ)" "$*"; }
run() { if [ "$DRY_RUN" -eq 1 ]; then log "DRY-RUN: $*"; else "$@"; fi; }

if [ "$DRY_RUN" -eq 0 ]; then
  read -r -p "Type FAILOVER to promote $SECONDARY_REGION: " answer
  [ "$answer" = "FAILOVER" ] || { log "Aborted by operator."; exit 1; }
fi

START=$(date +%s)

# 1. Secondary health
log "[1/6] Checking secondary region health"
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$SECONDARY_HEALTH_URL" || true)
[ "$code" = "200" ] || { log "FAIL: secondary unhealthy (HTTP ${code:-none})"; exit 1; }
log "  secondary healthy"

# 2. Replication lag within RPO (AuroraGlobalDBRPOLag, milliseconds)
log "[2/6] Reading replication lag from CloudWatch"
LAG_MS=$(aws cloudwatch get-metric-statistics \
  --region "$PRIMARY_REGION" --namespace AWS/RDS \
  --metric-name AuroraGlobalDBRPOLag \
  --dimensions Name=DBClusterIdentifier,Value="$SECONDARY_CLUSTER_ID" \
  --start-time "$(date -u -d '2 minutes ago' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" --period 60 --statistics Maximum \
  --query 'Datapoints[-1].Maximum' --output text)
[ "$LAG_MS" != "None" ] || { log "FAIL: no lag datapoints"; exit 1; }
log "  lag = ${LAG_MS} ms (budget ${RPO_MAX_MS} ms)"
awk -v lag="$LAG_MS" -v max="$RPO_MAX_MS" 'BEGIN{exit !(lag <= max)}' \
  || { log "FAIL: lag exceeds RPO budget"; exit 1; }

# 3. Promote secondary database (planned switchover; use failover-global-cluster
#    for unplanned recovery, promote-read-replica-db-cluster for non-global replicas)
log "[3/6] Switching over global cluster to secondary"
run aws rds switchover-global-cluster \
  --global-cluster-identifier "$GLOBAL_CLUSTER_ID" \
  --target-db-cluster-identifier "$SECONDARY_CLUSTER_ARN"
if [ "$DRY_RUN" -eq 0 ]; then
  aws rds wait db-cluster-available \
    --region "$SECONDARY_REGION" \
    --db-cluster-identifier "$SECONDARY_CLUSTER_ARN"
fi
log "  secondary cluster promoted"

# 4. Scale secondary compute
log "[4/6] Scaling secondary ASG to $ASG_DESIRED"
run aws autoscaling set-desired-capacity \
  --region "$SECONDARY_REGION" \
  --auto-scaling-group-name "$ASG_NAME" \
  --desired-capacity "$ASG_DESIRED"

# 5. Shift DNS
log "[5/6] Updating Route 53 record $DNS_RECORD"
CHANGE_BATCH=$(jq -n --arg n "$DNS_RECORD" --arg v "$SECONDARY_CNAME" '{
  Changes: [{Action: "UPSERT", ResourceRecordSet: {
    Name: $n, Type: "CNAME", TTL: 60,
    ResourceRecords: [{Value: $v}]}}]}')
run aws route53 change-resource-record-sets \
  --hosted-zone-id "$HOSTED_ZONE_ID" --change-batch "$CHANGE_BATCH"

# 6. Verify resolution and traffic
log "[6/6] Verifying DNS and traffic"
for i in 1 2 3 4 5 6; do
  resolved=$(dig +short "$DNS_RECORD" @1.1.1.1 | tail -1)
  [ "$resolved" = "$SECONDARY_CNAME" ] && break
  log "  attempt $i: got '${resolved:-nothing}', waiting 10s"
  sleep 10
done
[ "$resolved" = "$SECONDARY_CNAME" ] || { log "FAIL: DNS did not converge"; exit 1; }

code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "https://$DNS_RECORD/health" || true)
log "  public health: HTTP ${code:-none}"
log "=== Done in $(( $(date +%s) - START ))s — soak for 30 min before failback ==="

Reading the script

Steps 1 and 2 are the go/no-go gates in executable form: if the secondary can’t serve a health check or the replica is behind by more than the RPO budget, the script exits before changing anything. Step 3 uses switchover-global-cluster because a drill is a planned operation; the comments name the commands for the other two cases (unplanned failover-global-cluster, or promote-read-replica-db-cluster when the secondary is a standalone cross-region read replica rather than an Aurora Global member). Steps 4 and 5 size the compute and move DNS, and step 6 polls a public resolver until the record converges instead of trusting the API response.

Keep set -euo pipefail and the :? variable guards. They’re what turn a half-configured environment into a clean error at startup instead of a partial failover.

Failover Decision Flow

In practice, drills die on those two diamonds. The first is admitting the outage is real (people wait, hoping it resolves). The second is lag: every test plan says “check lag,” but the step only means something if someone has already decided what happens when the number is red.

Architecture Variants

Architecture	How failover works	RTO/RPO profile	Cost profile
Backup and restore	Rebuild region from snapshots	Hours / hours	Lowest
Pilot light	Core data replicates, compute off until needed	Tens of minutes / minutes	Low
Warm standby	Scaled-down copy always running	Minutes / seconds	Medium
Active-passive (hot)	Full secondary idle, promote on failure	Minutes / seconds	High
Active-active	Both serve traffic, shift on failure	Near zero / near zero	Highest
Cell-based	Users pinned to a cell, evacuate a cell	Minutes / seconds	Highest plus routing layer

The template’s timeline assumes warm standby or active-passive, which is where most teams land: a pilot light that takes 40 minutes to warm can’t meet a 15-minute RTO no matter how good the runbook is. If your service is cell-based, replace the DNS steps with cell evacuation; the go/no-go gates, fencing, evidence capture, and failback sections still apply unchanged.

Test Scenarios Worth Running

The happy path shouldn’t eat the whole drill. Rotate through these across quarterly tests:

1. Clean regional loss. Simulate a full outage (blackhole the primary ALB, or block it with a network ACL). Measures the baseline RTO and confirms the runbook order still works.
2. Degraded but alive. Throttle the primary to 20% success. This is the harder call: does 20% success justify a cutover? Teams without a written threshold either fail over too late or spend the window arguing.
3. Replication behind. Pause replication for a few minutes, then start the test. The lag check should block promotion until the replica catches up. If it doesn’t, your gate is decorative.
4. DNS surprise. Run the test the week after someone raises the TTL “temporarily.” Real propagation time replaces the assumed one in your RTO math.
5. Failback under load. Leave synthetic traffic running through the failback. Data written to the secondary during the soak has to survive the trip home; this is where row-count and checksum verification earn their place.
6. Missing person. Run the drill with the usual executor out. If the test can’t proceed without one specific engineer, that’s a finding to fix, not a success to log.

Capturing Evidence

The post-test report is the artifact an auditor or a post-incident review will ask for. During the test, the observer logs wall-clock times for each timeline step, the lag value recorded at promotion, DNS propagation measurements from at least two resolvers, and error rates during the cutover window. Attach dashboard screenshots and the script’s output log to the completed document.

Compare actuals against targets honestly. A test that misses RTO by four minutes and documents why is a better outcome than one that hits it because everyone agreed not to look at the clock too closely. Convert every surprise into an action item with an owner and a date; the action items are the return on the drill’s cost.

What Works

• Run the drill in production at least quarterly. Staging tests verify the script; only production verifies the load path, real DNS, and real traffic.
• Keep the secondary warm enough to meet RTO. If the secondary needs 40 minutes to turn useful, your 15-minute RTO is gone before the drill even begins.
• Write the abort criteria down before the window, and make sure the person holding abort authority knows they hold it.
• Test failback every time. The return path is where untested replication direction, forgotten sequence states, and split-brain bugs live.
• Automate detection and evidence collection; keep the promotion decision human until the false-positive rate of your signals is low enough to trust.

Common Mistakes

• Treating failover as DNS-only. Traffic shifting is the last step; data, compute, and write fencing come first, in that order.
• Leaving the failover record at a 3600-second TTL until test day and discovering the RTO was never reachable.
• Promoting before fencing, which is how a drill creates the split-brain it was supposed to prevent.
• Skipping the soak. Declaring victory after the first 200 response misses the cache misses, expired regional secrets, and missing queue consumers that only real traffic exposes.
• Reusing the same scenario every quarter, so the team gets good at the drill instead of good at failover.
• Filing the report without action items. An undocumented surprise will be rediscovered during the next real outage.

Troubleshooting

• Replica won’t promote. On Aurora Global Database, a secondary can’t be promoted to a headless cluster target until there’s at least one DB instance in it, and switchover requires matching engine versions between primary and target. Check describe-global-clusters output for cluster status before retrying; a cluster stuck in modifying won’t accept the call.
• DNS updated but traffic still lands on the primary. Almost always a cached answer: a TTL above 60, a resolver honoring stale records, or a CDN layer with its own origin TTL in front of Route 53. Measure with dig +trace and against public resolvers, and check any CDN or global accelerator sitting between clients and DNS.
• Replication lag grows after failover. The new primary is now feeding replicas the other direction, and undersized secondary capacity (instance class, IOPS, network) shows up as lag during write peaks. The soak is where lag surprises show up, so keep watching it for the whole window, not only at promotion.
• Health checks pass but users see errors. Regional differences: a missing secret in the secondary’s parameter store, an IAM policy that doesn’t exist in region B, or a downstream dependency that was never replicated. Keep a per-region parity checklist and diff it before each drill.
• Failback leaves stale writes. If anything wrote to the old primary after fencing, those writes aren’t in the new primary’s history. Reconciliation means identifying divergent records by timestamp or transaction log and replaying or discarding them deliberately; log the decision in the report.
• The script exits early. Read which gate failed. A health-check failure means fix the secondary first; a lag failure means wait or accept the loss explicitly. Never bypass a gate by editing the script mid-test.

Keeping the Template Current

Assign the document an owner and a review date. Review it after every drill and every relevant incident, retire steps the infrastructure outgrew, and re-fill the timeline budgets when measured times drift. A template that describes last year’s stack is worse than no template, because it lends false confidence to the person holding it.

Further Reading

• Using switchover or failover in Amazon Aurora Global Database — managed switchover (RPO 0) versus unplanned failover, plus the lag metrics to watch.
• Creating Amazon Route 53 health checks and configuring DNS failover — health-check-driven routing so DNS shifting isn’t a manual step.
• Amazon Application Recovery Controller (ARC) — readiness checks and routing controls for multi-region recovery.
• Google Cloud disaster recovery planning guide — the same RTO/RPO patterns mapped to GCP services.
• The Database Failover Runbook covers engine-level promotion detail (PostgreSQL, MySQL, Patroni) that this template deliberately leaves at the region level.