Backup Verification Test Template

Overview

A backup that cannot be restored is not a backup. This template helps teams schedule, execute, and document backup verification tests. It covers the systems under test, the restore procedure, validation criteria, and what to do when a test fails. Related resources: Go REST API with Gin and Middleware, Build gRPC Services in TypeScript with Protocol Buffers, and Schedule Periodic Tasks in Python with APScheduler.

When to Use

• For alternatives, see Disaster Recovery: RTO, RPO, and Resilient Recovery Runbooks.

• After configuring a new backup policy or tool.

• Before a compliance audit or disaster recovery review.

• After a production restore incident revealed gaps.

• On a recurring schedule (monthly, quarterly, or yearly depending on criticality).

• When recovery time objective (RTO) or recovery point objective (RPO) requirements change.

Prerequisites

• Documented backup policy and retention schedule.
• Access to backup storage and target restore environment.
• A maintenance window or isolated test environment that does not affect production.
• Owner for each system being tested.
• Defined RTO and RPO targets for each workload.
• A method to validate restored data and application behavior.

Solution

Template

1. Test Identification

Field	Description	Example
Test ID	Unique identifier	BVT-2026-Q3-001
System / Application	What is being tested	Customer database
Environment	Where the restore is tested	Isolated DR sandbox
Backup Type	Full, incremental, snapshot, object copy	Nightly snapshot
Backup Date	Point in time of the backup	2026-06-25 02:00 UTC
Test Owner	Person responsible for execution	SRE team
Scheduled Date	When the test is performed	2026-06-27
Stakeholders	Teams to notify	DBA, security, application team

2. Scope and Objectives

Objective	Target	Measurement
Verify backup integrity	Restore completes without corruption	Hash match or application health check
Validate RTO	Restore within agreed time	Compare elapsed time to RTO
Validate RPO	Data loss within agreed window	Compare backup age to RPO
Confirm dependencies	Required services and credentials available	Checklist passed
Test runbook accuracy	Steps produce expected outcome	No deviations logged

3. Restore Procedure

Step	Action	Expected Result	Actual Result	Pass / Fail
1	Identify backup media and location	Backup found and accessible
2	Provision target restore environment	Environment ready and isolated
3	Copy backup to target	Transfer completes without errors
4	Execute restore command	Restore completes successfully
5	Verify file system or database state	All expected objects present
6	Start application services	Services reach healthy state
7	Run validation checks	Smoke tests pass
8	Capture logs and metrics	Evidence collected
9	Clean up test environment	Resources removed

4. Validation Checklist

• Restored data size matches backup size (within expected tolerance).
• No corruption errors reported by restore tool or checksum validation.
• Application can connect to restored database or storage.
• Critical read queries or file reads return expected results.
• Write operations can be performed in the test environment without affecting production.
• RTO is met or a documented exception is recorded.
• RPO is met or a documented exception is recorded.
• Credentials, secrets, and network access work after restore.
• Logs show no unexpected errors during the restore.
• Runbook steps are accurate and complete.

5. Results Summary

Metric	Target	Actual	Status
Restore duration	< 60 minutes	47 minutes	Pass
Data freshness	< 4 hours	3 hours	Pass
Application smoke tests	100% pass	100% pass	Pass
Runbook accuracy	No deviations	2 minor deviations	Pass with notes
Aggregate test result	Pass		Pass

6. Issue Log and Remediation

Issue ID	Description	Severity	Owner	Due Date	Status
BVT-001	Restore script uses hard-coded path	Medium	SRE team	2026-07-04	Open
BVT-002	Documentation missing step for secret rotation	Low	Platform team	2026-07-11	Open

Explanation

Backup verification is the only way to prove that a disaster recovery plan works. Regular tests expose issues like missing backups, credential drift, runbook errors, and RTO/RPO mismatches before an emergency. Documenting each test creates an audit trail and drives continuous improvement of restore procedures.

PostgreSQL Restore Verification Script

#!/bin/bash
# Restore and verify a PostgreSQL backup
set -euo pipefail

BACKUP_FILE="/backups/prod_db_2026-07-11.sql.gz"
TEST_DB="restore_test_$(date +%s)"
PG_HOST="test-db.internal"
PG_USER="restore_verifier"

echo "=== PostgreSQL Backup Restore Verification ==="
echo "Backup: $BACKUP_FILE"
echo "Test DB: $TEST_DB"
echo ""

# Create test database
echo "[1/6] Creating test database..."
createdb -h "$PG_HOST" -U "$PG_USER" "$TEST_DB"

# Restore backup
echo "[2/6] Restoring backup..."
gunzip -c "$BACKUP_FILE" | psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -v ON_ERROR_STOP=1 > /dev/null

# Verify row counts
echo "[3/6] Verifying row counts..."
TABLES=$(psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -t -c "SELECT tablename FROM pg_tables WHERE schemaname='public'")
for table in $TABLES; do
  count=$(psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -t -c "SELECT count(*) FROM $table")
  echo "  $table: $count rows"
done

# Verify constraints
echo "[4/6] Verifying constraints..."
CONSTRAINTS=$(psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -t -c "SELECT count(*) FROM pg_constraint WHERE conrelid IN (SELECT oid FROM pg_class WHERE relnamespace='public'::regnamespace)")
echo "  Active constraints: $CONSTRAINTS"

# Verify indexes
echo "[5/6] Verifying indexes..."
INDEXES=$(psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -t -c "SELECT count(*) FROM pg_indexes WHERE schemaname='public'")
echo "  Active indexes: $INDEXES"

# Run test queries
echo "[6/6] Running smoke queries..."
psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -c "SELECT 1 as test" > /dev/null && echo "  Smoke query: PASS" || echo "  Smoke query: FAIL"

# Measure restore time
RESTORE_TIME=$(psql -h "$PG_HOST" -U "$PG_USER" -d "$TEST_DB" -t -c "SELECT now() - '$START_TIME'::timestamp")

# Cleanup
echo ""
echo "Cleaning up test database..."
dropdb -h "$PG_HOST" -U "$PG_USER" "$TEST_DB"
echo "=== Verification Complete ==="

RTO/RPO Measurement Worksheet

=== Backup Verification RTO/RPO Worksheet ===

Test Date: 2026-07-11
Service: production-database
Backup Type: Full + WAL streaming

RPO Measurement:
  - Last successful backup: 2026-07-11 02:00 UTC
  - Last WAL archived:     2026-07-11 10:45 UTC
  - Test restore point:    2026-07-11 11:00 UTC
  - Data loss:             15 minutes
  - RPO Target:            30 minutes
  - RPO Status:            PASS (15 min < 30 min)

RTO Measurement:
  - Restore start time:    11:00 UTC
  - Database available:    11:08 UTC
  - Application connected: 11:10 UTC
  - Smoke tests passed:    11:12 UTC
  - Total RTO:             12 minutes
  - RTO Target:            30 minutes
  - RTO Status:            PASS (12 min < 30 min)

Issues Found:
  - WAL archive gap of 3 minutes during 09:30-09:33
  - Restore script hard-coded path (BVT-001)
  - Missing secret rotation step (BVT-002)

Remediation:
  - Investigate WAL gap cause
  - Fix hard-coded paths by 2026-07-04
  - Add secret rotation to runbook by 2026-07-11

Variants

• Database backup verification: Restore full and incremental backups, verify transaction log replay, and run consistency checks.
• File system backup verification: Restore directories, validate permissions, and compare checksums.
• Virtual machine backup verification: Boot restored VM, verify network and services, then run application tests.
• Object storage backup verification: Restore selected objects, validate metadata, and compare against source bucket.
• Cloud snapshot verification: Create a temporary volume from snapshot, mount it, and validate data integrity.
• Application-level backup verification: Restore data into a fresh application instance and run end-to-end smoke tests.

What Works

• Test backups on a recurring schedule, not just once a year.
• Use an isolated environment that mirrors production topology.
• Automate restore steps where possible, but keep a manual runbook.
• Validate both data integrity and application behavior after restore.
• Measure and compare actual RTO/RPO against targets every time.
• Record deviations and remediate before the next test.
• Rotate credentials and secrets in test environments to match production.
• Keep backup metadata accessible without relying on the production system.
• Include backup verification in change management for critical systems.
• Store test evidence for compliance and audits.

Common Mistakes

• Assuming a backup is valid because the backup job reported success.
• Testing only full backups and ignoring incremental or differential chains.
• Restoring to the same environment where the backup was taken.
• Skipping application validation after data restore.
• Not testing credential or network dependency restoration.
• Failing to document and fix issues found during tests.
• Testing too infrequently to catch configuration drift.
• Ignoring backup size growth and restore time trends.

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the backup and disaster-recovery guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply backup verification test template when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Common Production Pitfalls

• Leaving required fields blank or using vague one-word answers.
• Filling the document once and never updating it after scope or decisions change.
• Storing the document where the team does not look during incidents or reviews.
• Not assigning an owner, due date, or review cadence.
• Copying boilerplate without removing sections that do not apply.
• Skipping version control, which makes rollback and accountability impossible.
• Failing to link the document to related decisions or follow-up actions.
• Avoiding quarterly reviews that would retire stale or unused sections.