Deployment Rollback Runbook

Overview

When a deployment breaks production you have minutes, not hours. This runbook gives you the decision criteria for when to roll back, the exact commands for each deployment strategy (kubectl, Helm, ArgoCD, blue-green, canary), the database migration rollback paths, and the verification and communication steps for after the rollback. Every procedure is copy-paste ready — during an incident you should be executing, not improvising.

This runbook covers the operational procedure of reverting a bad deployment. For planning the deployment strategy itself, see Blue-Green and Canary Deployments; for migration-heavy incidents, see the Data Migration Runbook.

When to Use

Reach for this runbook when a deployment degrades production and you need the fastest safe path back to a known-good state:

• Error rate or latency spiking right after a deploy
• Health checks failing on new pods
• A migration broke the application contract
• A canary is showing negative metrics and needs aborting

Do not roll back for config-only changes (fix the ConfigMap and redeploy), for a feature that can be disabled with a flag (toggle it off instead), or when the “bad” deploy only exposed a pre-existing bug (fix-forward is faster and keeps history linear).

One judgment call before the table: rollback is not always the right answer. If the bad deploy is a small, well-understood bug and a fix is 15 minutes away, fix-forward can be the lower-risk move — you avoid bouncing production twice. Roll back when the failure is broad, the cause is unclear, or users are actively impacted; fix-forward when the defect is isolated and you can verify the fix fast. When in doubt during a live incident, roll back — you can always deploy again once it’s fixed.

Pick the rollback path that matches how the change shipped:

Deployment method	Rollback path	Reverts
kubectl apply / plain Deployment	kubectl rollout undo (§2)	Pod template only — not Service, Ingress, or ConfigMap changes
Helm release	helm rollback (§3)	All resources tracked by the release revision
ArgoCD / GitOps	Git revert (§4.2)	Whatever the reverted commit changed — cleanest audit trail
Blue-green	Traffic switch back (§5)	Routing only; both versions stay deployed
Canary (Argo Rollouts / manual)	Abort or scale-to-zero (§6)	Traffic split; stable version unaffected
Database migration	Fix-forward / down migration / snapshot (§7)	Schema and data — the slowest and riskiest path

Before You Start

Thirty seconds of verification beats a second failed deployment:

• Confirm your kubectl context and namespace — kubectl config current-context. Rolling back the wrong cluster is worse than not rolling back.
• Check you have permissions — you need update/patch on deployments and, for Helm, access to the release secrets. Find out now, not mid-incident.
• Know your last good version — tag releases or keep revisionHistoryLimit ≥ 10 on Deployments so the revision you need still exists.
• Take a database snapshot before risky deploys — if the release includes a migration, a pre-deploy snapshot is the only real undo for destructive changes.
• Disable auto-sync on GitOps-managed apps — if ArgoCD auto-syncs while you’re reverting, it will happily redeploy the bad state.

1. Rollback Triggers

Rollback is a judgment call under pressure. The trigger table exists so that call is made before the incident, not during it — agree on thresholds with your team ahead of time and encode them into alerting where possible.

1.1 Trigger Criteria

Trigger                    | Severity | Action              | Timeline
───────────────────────────┼──────────┼─────────────────────┼──────────
Error rate > 5%            | Critical | Rollback immediately | < 5 min
Error rate > 1%            | High     | Investigate, prepare | < 15 min
P99 latency > 2x baseline  | High     | Rollback if trending | < 15 min
P99 latency > 5x baseline  | Critical | Rollback immediately | < 5 min
Health check failures      | Critical | Rollback immediately | < 5 min
OOM kills increasing       | High     | Rollback if trending | < 10 min
Customer complaints > 10   | High     | Investigate, prepare | < 15 min
Deployment job timeout     | Medium   | Investigate          | < 30 min
Database connection errors | Critical | Rollback immediately | < 5 min

1.2 Rollback Decision Tree

The tree is deliberately aggressive: rolling back a deploy that turned out to be fine costs you a redeploy; not rolling back a bad one costs you an outage.

2. Kubernetes Deployment Rollback

kubectl rollout undo reverts the Deployment’s pod template to a previous revision. It does not revert changes to Services, Ingresses, ConfigMaps, or Secrets applied in the same release — those need their own kubectl apply with the previous manifest.

Before you start, make sure .spec.revisionHistoryLimit on the Deployment is high enough to hold the revision you want (default is 10; if you deploy often, the good revision may already be garbage-collected).

2.1 kubectl Rollback

# Check rollout history
kubectl rollout history deployment/my-app -n production

# Check details of a specific revision
kubectl rollout history deployment/my-app -n production --revision=3

# Rollback to previous revision
kubectl rollout undo deployment/my-app -n production

# Rollback to specific revision
kubectl rollout undo deployment/my-app -n production --to-revision=3

# Check rollout status
kubectl rollout status deployment/my-app -n production

# Pause rollout (if canary and need to stop)
kubectl rollout pause deployment/my-app -n production

# Resume rollout
kubectl rollout resume deployment/my-app -n production

2.2 Verify Rollback

# Check current image version
kubectl get deployment my-app -n production -o jsonpath='{.spec.template.spec.containers[*].image}'

# Check pod status
kubectl get pods -n production -l app=my-app -o wide

# Check pod logs for errors
kubectl logs deployment/my-app -n production --tail=50

# Check events for deployment issues
kubectl get events -n production --field-selector involvedObject.name=my-app --sort-by='.lastTimestamp'

# Run health check
kubectl exec -it deployment/my-app -n production -- curl -s http://localhost:8080/health

3. Helm Rollback

helm rollback restores the full release to a previous revision — unlike kubectl rollout undo, it reverts every resource the release tracks, including Services and ConfigMaps. It still won’t touch things outside the release: CRDs installed separately, data written to volumes, or external resources like DNS records.

3.1 Helm Rollback Commands

# List Helm releases
helm list -n production

# Check release history
helm history my-app -n production

# Rollback to previous revision
helm rollback my-app -n production

# Rollback to specific revision
helm rollback my-app 5 -n production

# Rollback with timeout
helm rollback my-app 5 -n production --timeout 5m

# Verify rollback
helm status my-app -n production
kubectl get pods -n production -l app.kubernetes.io/instance=my-app

3.2 Helm Rollback with Cleanup

A failed rollback usually means stuck resources, not a broken Helm — check for pods that won’t terminate and for release secrets left in a bad state:

# If rollback fails, check for stuck resources
kubectl get all -n production -l app.kubernetes.io/instance=my-app

# Force delete stuck pods
kubectl delete pod <pod-name> -n production --force --grace-period=0

# Check for pending PVCs
kubectl get pvc -n production -l app.kubernetes.io/instance=my-app

# Clean up failed Helm secrets
kubectl get secrets -n production -l owner=helm,name=my-app
kubectl delete secret sh.helm.release.v1.my-app.v6 -n production

4. ArgoCD Rollback

With GitOps there are two paths: the CLI rollback (fast, imperative) and the Git revert (slower, but keeps the desired state in Git — which is the whole point of GitOps). Prefer the Git revert when the immediate pressure is manageable.

Note: argocd app rollback is imperative — it changes live state without touching Git. With auto-sync enabled, Argo CD will detect the drift and redeploy the newer (bad) state, so disable auto-sync first and follow up with a Git revert as soon as the incident is under control.

4.1 ArgoCD CLI Rollback

# Get application status
argocd app get my-app

# Check sync history
argocd app history my-app

# Rollback to previous sync
argocd app rollback my-app

# Rollback to specific revision
argocd app rollback my-app 5

# Disable auto-sync before rollback (if enabled)
argocd app set my-app --sync-policy none

# Perform rollback
argocd app rollback my-app 5

# Re-enable auto-sync after rollback
argocd app set my-app --sync-policy automated --auto-heal

4.2 ArgoCD Git-based Rollback

# Git-based rollback — revert the commit and push
git revert <bad-commit-sha>
git push origin main

# ArgoCD detects the change and syncs automatically (if auto-sync enabled)
# If auto-sync disabled, manually sync:
argocd app sync my-app

# Force sync if needed
argocd app sync my-app --force

5. Blue-Green Deployment Rollback

Blue-green rollback is a routing change, not a redeploy — the old environment is still running, so reverting is close to instant. That is the main reason to pay the double-infrastructure cost. One precondition to check first: the blue environment must actually still be there. If your pipeline tears down the previous color after a successful switch, the rollback becomes a redeploy of the old version — slower, but still safe.

5.1 Blue-Green Switch

# Current state: blue is active, green is new deployment
# To rollback: switch traffic back to blue

# Kubernetes service selector switch
kubectl patch service my-app -n production -p \
  '{"spec":{"selector":{"version":"blue"}}}'

# Verify traffic switched
kubectl get svc my-app -n production -o yaml | grep selector -A 3

# Check that pods are receiving traffic
kubectl get pods -n production -l version=blue -o wide

# Scale down green deployment (after confirming blue is healthy)
kubectl scale deployment my-app-green -n production --replicas=0

5.2 Istio VirtualService Rollback

# Rollback: route 100% traffic back to blue
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: my-app
  namespace: production
spec:
  http:
    - route:
        - destination:
            host: my-app-blue
            port:
              number: 8080
          weight: 100

# Apply the rollback VirtualService
kubectl apply -f virtualservice-rollback.yaml -n production

# Verify traffic routing
kubectl get virtualservice my-app -n production -o yaml

6. Canary Rollback

Canary rollback is a traffic decision: stop shifting traffic to the new version and let the stable version take 100% again. With Argo Rollouts the controller does it for you when the analysis run fails — define error-rate and latency thresholds in an AnalysisTemplate and the abort becomes automatic rather than a pager decision. Without automated analysis, the procedure below is manual and the trigger table from section 1 is your abort criteria.

6.1 Argo Rollouts Canary Rollback

# Check rollout status
kubectl argo rollouts get rollout my-app -n production --watch

# Abort canary and rollback to stable
kubectl argo rollouts abort my-app -n production

# Promote canary to stable (if canary is good)
kubectl argo rollouts promote my-app -n production

# Retry rollout after abort
kubectl argo rollouts retry my-app -n production

6.2 Manual Canary Rollback

# Current state: 20% traffic on new version, 80% on stable
# To rollback: scale new version to 0, restore stable to 100%

# Scale down canary deployment
kubectl scale deployment my-app-canary -n production --replicas=0

# Scale up stable deployment
kubectl scale deployment my-app-stable -n production --replicas=10

# Update service to point only to stable
kubectl patch service my-app -n production -p \
  '{"spec":{"selector":{"track":"stable"}}}'

# Verify
kubectl get endpoints my-app -n production
kubectl get pods -n production -l track=stable

7. Database Migration Rollback

Database rollbacks are the slowest and riskiest part — code reverts in seconds, data doesn’t. Whether you can roll back at all depends on how the migration was designed, so the real work happens before the deployment: prefer expand-contract for anything that drops or renames columns.

For migration-heavy incidents beyond a single schema change, use the Data Migration Runbook.

Order matters when app and schema change together: if the old application version can’t run against the new schema, roll the database back first (or you’ll take the healthy old version down too). If the new schema is backward-compatible — which expand-contract gives you by design — roll the app back first and fix the schema later without pressure.

7.1 Migration Rollback Strategy

Migration type        | Rollback strategy
──────────────────────┼──────────────────────────────────────────
Forward-only          | No rollback — write fix-forward migration
Reversible            | Run down migration (reverse of up)
Expand-contract       | Revert contract phase, keep expand changes
Snapshot restore      | Restore from backup (last resort, data loss)

7.2 Flyway Rollback

# Check migration status
flyway -url=jdbc:postgresql://db:5432/mydb info

# Undo last migration (Flyway Teams only)
flyway -url=jdbc:postgresql://db:5432/mydb undo

# For community edition — write a fix-forward migration
# Create V20260704_2__rollback_add_column.sql

-- Fix-forward migration to undo a bad change
-- V20260704_1__add_status_column.sql added a column that broke the app
-- V20260704_2__remove_status_column.sql reverts it

ALTER TABLE orders DROP COLUMN IF EXISTS status;

7.3 Liquibase Rollback

# Check migration status
liquibase --url=jdbc:postgresql://db:5432/mydb status

# Rollback by count (last N changesets)
liquibase --url=jdbc:postgresql://db:5432/mydb rollbackCount 1

# Rollback by tag
liquibase --url=jdbc:postgresql://db:5432/mydb rollback v2.3.0

# Rollback by date
liquibase --url=jdbc:postgresql://db:5432/mydb rollbackToDate 2026-07-04

7.4 Expand-Contract Pattern

Phase 1 — Expand (add new column, keep old)
  ALTER TABLE orders ADD COLUMN status_new VARCHAR(20);

Phase 2 — Migrate (dual-write to both columns)
  -- Application writes to both status and status_new
  -- Backfill: UPDATE orders SET status_new = status WHERE status_new IS NULL;

Phase 3 — Contract (switch reads to new column, remove old)
  -- Application reads from status_new
  ALTER TABLE orders DROP COLUMN status;
  ALTER TABLE orders RENAME COLUMN status_new TO status;

Rollback:
  - After Phase 1: DROP COLUMN status_new (no data loss)
  - After Phase 2: DROP COLUMN status_new (old column still intact)
  - After Phase 3: Cannot rollback — must re-add column and backfill

7.5 Snapshot Restore (Last Resort)

Snapshot restore means data loss for everything written after the snapshot — measure that window before you pull the trigger:

# Restore from pre-deployment snapshot
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier myapp-db-rolled-back \
  --db-snapshot-identifier pre-deploy-snapshot-2026-07-04

# Or use point-in-time recovery
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier myapp-db \
  --target-db-instance-identifier myapp-db-rolled-back \
  --restore-time 2026-07-04T09:00:00Z

8. Post-Rollback Procedures

The rollback isn’t done when the old version is serving — it’s done when you’ve verified health, told stakeholders, and captured what you need for the post-mortem. Skipping the verification checklist is how teams end up “rolling back” to a version that was broken in a different way.

8.1 Verification Checklist

- [ ] Application health check passes
- [ ] Error rate returns to baseline (< 0.1%)
- [ ] P99 latency returns to baseline
- [ ] All pods running and ready
- [ ] Database connections healthy
- [ ] No new OOM kills
- [ ] Monitoring dashboards show normal patterns
- [ ] Customer complaints stop or decrease
- [ ] Logs show no new errors
- [ ] Alerting returns to normal state

8.2 Post-Rollback Actions

1. Verify rollback using the checklist above
2. Notify stakeholders (Slack, email, status page)
3. Create incident ticket if not already created
4. Capture timeline of events (deploy time, detection, rollback)
5. Preserve logs and metrics for post-mortem
6. Do NOT re-deploy the same version without a fix
7. Identify root cause of the failure
8. Write fix and test in staging
9. Schedule post-mortem meeting within 48 hours
10. Update deployment runbook with lessons learned

8.3 Communication Template

[RESOLVED] Production deployment rollback — my-app

Timeline:
  - 14:00 UTC: Deployment v2.3.1 started
  - 14:05 UTC: Error rate increased to 8%
  - 14:07 UTC: Rollback initiated
  - 14:10 UTC: Rollback complete, error rate back to 0.1%

Impact:
  - Users experienced 500 errors for approximately 10 minutes
  - ~5% of requests failed during the incident
  - No data loss or corruption

Root cause (preliminary):
  - Database connection pool misconfiguration in v2.3.1

Action items:
  - Fix connection pool configuration
  - Add pre-deployment database connection test
  - Update CI pipeline to catch this configuration error

Current status:
  - Production is running v2.3.0 (previous stable version)
  - All services are healthy
  - Next deployment scheduled after fix is verified in staging

See Also

• Complete Guide to GitOps with ArgoCD
• Helm Charts: Structure, Templating, Dependencies, Registry
• Blue-Green and Canary Deployments
• Canary Deployments with Istio Service Mesh
• Package Kubernetes Manifests with Helm Charts
• Kubernetes Deployments — official docs
• Helm rollback — official docs
• Argo CD — official docs