Zero-Downtime Deployment Checklist

Overview

A single bad deploy can take a revenue-critical service offline in seconds. Zero-downtime deployments update production while users stay connected, but they only work when health checks actually gate traffic, database changes stay compatible with both code versions, and rollback is a tested procedure instead of a hope. This checklist covers the full release window: what to verify before you ship, how to route traffic during the rollout, and the exact signals that should trigger a rollback.

It focuses on deployments that must not interrupt traffic. For a general pre-release verification list, see the Deployment Checklist Template.

The checklist assumes Kubernetes or a comparable orchestrator, but the same gates apply to any platform: prove the new version is healthy before it receives traffic, keep the old version alive until the new one has earned full traffic, and make the rollback path shorter than the time it takes users to notice an outage.

When to Use

• Releasing a new version of a user-facing service where dropped requests cost revenue or trust.
• Deploying schema or data migrations while old and new code run side by side.
• Changing load balancers, ingress rules, or other infrastructure that can cut availability.
• Introducing a rollout strategy such as canary or blue-green for the first time.
• Shipping ahead of a traffic peak where a partial outage would be visible.

Don’t use it for a batch job or internal tool where a maintenance window is acceptable, or for destructive schema changes that were never designed to be backward compatible — fix the migration strategy first.

Prerequisites

• A CI/CD pipeline that produces a tagged, immutable artifact without manual steps.
• Health check endpoints that report real dependency status, not just process liveness.
• A load balancer, ingress, or service mesh that supports gradual traffic shifting.
• Database migrations written to be backward compatible (expand-contract).
• A documented rollback path with a known-good artifact and data state.
• Dashboards and alerts covering error rate, latency percentiles, and business metrics.
• An agreed communication channel and on-call owner for the release window.

The Checklist

Work through the seven phases in order — each one assumes the previous phase passed. The checklist is designed to be printed or pasted into a release ticket; every unchecked box is a deliberate decision you should be able to defend in a postmortem.

1. Pre-Deployment Readiness

• The change is approved and documented, with a named deployer and on-call owner.
• Code is merged and the artifact is built, tagged, and immutable.
• Unit, integration, and contract tests pass in CI.
• Database migrations were reviewed for backward compatibility.
• Feature flags are configured so new behavior can be toggled off without redeploying.
• Capacity covers the expected traffic plus the surge from duplicated instances during rollout.
• Dashboards and alerts are live and linked in the release ticket.
• The on-call rotation knows the deployment window and the escalation path.
• Rollback steps were tested in staging within the last quarter, not just written down.
• Customer-facing communication is drafted if the change is user-visible.

2. Health Check Configuration

Check	Endpoint	Success Criteria	Failure Action
Liveness	/health/live	HTTP 200	Restart container
Readiness	/health/ready	HTTP 200 and dependencies reachable	Stop traffic routing
Startup	/health/startup	HTTP 200	Delay rollout
Dependency	/health/deps	Database, cache, and queue respond	Alert and halt
Business	/health/business	Critical flow returns expected value	Page on-call

A readiness probe that returns 200 without checking the database is the single most common cause of “successful” deployments that immediately fail for users. The probe should fail when a dependency is down, even if the process is healthy.

3. Rollout Strategy Selection

Strategy	Use Case	Risk Level	Rollback Speed
Rolling update	Stateless services, low risk	Low	Medium (terminate new pods)
Blue-green	Stateful sessions, predictable releases	Medium	Fast (switch traffic back)
Canary	High risk, measurable metrics	Medium	Fast (drain canary)
Feature flag	Gradual user exposure	Low	Instant (toggle off)
A/B deployment	Validate user behavior	Medium	Fast (re-route traffic)

Pick canary when you can define a metric gate, blue-green when sessions or caches make mixed-version traffic risky, and rolling updates for low-risk stateless services. The Canary Deployment Guide covers metric gating in depth; for traffic splitting with Istio see Istio Canary Deployment.

4. Deployment Execution Steps

Step	Action	Verification
1	Deploy to staging and run smoke tests	Staging tests pass
2	Deploy canary or a small subset	Health checks pass, error rate stable
3	Monitor key metrics for the canary duration	Latency, error rate, business metrics within baseline
4	Increase traffic percentage gradually	Each stage passes health and metric checks
5	Complete rollout to 100%	All instances healthy and serving traffic
6	Validate production endpoints	Smoke tests and critical user flows pass
7	Keep the old version available for rollback	Retain for the defined rollback window
8	Confirm the rollback window has passed	Remove old version or update the artifact baseline

5. Database Migration Safety

• Migrations are additive and work with the previous application version.
• Old code can read the new schema without errors.
• New code can read the old schema if a rollback is needed.
• Indexes are created concurrently where the engine supports it.
• Large migrations are split into batches small enough to stay under lock timeouts.
• Backfill and migration jobs are idempotent and resumable.
• A rollback script or compensating operation exists and was tested.
• Schema changes were tested in staging against production-like data volume.

6. Rollback Triggers

Trigger	Threshold	Action
Error rate spike	> 0.5% for 2 minutes	Pause rollout and investigate
Latency increase	p99 > baseline + 30% for 5 minutes	Roll back traffic
Business metric drop	Conversion rate drops > 5%	Roll back immediately
Health check failure	> 10% of instances failing	Roll back immediately
Critical alert	Any P1 incident	Roll back and page on-call
Canary timeout	Canary stage exceeds its duration without passing	Roll back the canary

Tie these thresholds to your SLOs rather than copying them blindly. A service with a 99.9% availability target can tolerate a shorter error budget burn than these defaults. For the rollback procedure itself, keep the Deployment Rollback Runbook open during the release.

7. Post-Deployment Validation

• Application logs show no unexpected errors or new exception types.
• Error rate and latency sit within baseline for at least 30 minutes.
• Business metrics are stable or improving.
• Feature flags are in their intended state.
• Old resources stay available until the rollback window closes, then get cleaned up.
• A deployment summary goes out to the team with links to dashboards.
• Any issues found are logged in the tracker with owners.

How It Works

Zero-downtime deployment stands on three pillars. Safe rollout mechanics keep old and new versions serving traffic at the same time. Reliable health signals tell the traffic controller when a new instance is genuinely ready. Fast rollback shrinks the blast radius when either of the first two fails. The checklist exists because the failure mode is always the same: a step that “usually works” gets skipped under time pressure, and the one release where it mattered takes the service down.

The economics favor patience. A canary at 10% that catches a regression exposes it to a tenth of your users for a few minutes. The same regression shipped to 100% of traffic at once turns into a full outage, a longer mean time to detect, and a rollback performed under incident pressure — exactly the conditions where mistakes compound. Measured in total user-visible downtime, the staged path is the fast one.

Kubernetes Rolling Update Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v2.3.1
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 15 && kill -SIGTERM 1"]
      terminationGracePeriodSeconds: 60

maxUnavailable: 0 keeps every replica serving while maxSurge: 3 spins up new pods alongside them. The preStop hook sleeps before SIGTERM so the kubelet has time to remove the pod from endpoints — without it, in-flight requests hit a pod that’s already shutting down.

Argo Rollouts Canary Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-gateway
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api-gateway
        - setWeight: 30
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api-gateway
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v2.3.1
          ports:
            - containerPort: 8080

Each analysis step runs an AnalysisTemplate (here success-rate) against your metrics provider; a failed analysis aborts the rollout and shifts traffic back automatically. Adjust weights and pauses to your traffic volume — a canary at 10% needs enough requests per minute for the metrics to be statistically meaningful.

Canary Rollout Flow

Connection Draining and DNS

Two details silently break otherwise clean rollouts. First, connection draining: when the load balancer removes an instance, keep terminationGracePeriodSeconds above the longest request you expect — 60 seconds works for typical HTTP APIs but not for WebSocket or streaming endpoints, where you should send a server-side close frame and wait for the active connection count to reach zero. On AWS set the target group deregistration delay to match; on GCP use the backend service connection draining timeout.

Second, DNS. If your blue-green switch relies on DNS, lower the record TTL to 30-60 seconds at least a day before cutover — resolvers cache aggressively and an “instant” switch can take the old TTL’s lifetime for some clients. Prefer weighted routing at the load balancer or mesh layer where possible so the switch doesn’t depend on client behavior.

Database Migrations with Expand-Contract

Expand-contract splits a breaking schema change into three safe deployments:

-- Phase 1 (expand): add the new shape alongside the old one
ALTER TABLE orders ADD COLUMN total_cents BIGINT;
CREATE INDEX CONCURRENTLY idx_orders_total_cents ON orders(total_cents);

-- Phase 2 (migrate): dual-write from the app, backfill in batches
UPDATE orders SET total_cents = ROUND(total * 100)
 WHERE id BETWEEN :lo AND :hi AND total_cents IS NULL;

-- Phase 3 (contract): a later deploy drops the old column
ALTER TABLE orders DROP COLUMN total;

The rules that make this safe: never rename or drop a column in the same release that deploys the new code, keep backfills idempotent and batched so a failed job can resume, and create indexes CONCURRENTLY on Postgres to avoid table locks. Run each phase as a separate deployment with its own validation window.

Minute-by-Minute Canary Runbook

Time	Action	Gate
T-0	Deploy canary at 10% weight	Pods ready, no error spike
T+5m	Review error rate, p99, business metrics	All within baseline
T+10m	Scale to 30%	Analysis passes
T+15m	Review metrics again	All within baseline
T+20m	Scale to 50%	Analysis passes
T+30m	Scale to 100%	Analysis passes
T+60m	Post-deploy validation checklist	All items checked
T+24h	Close rollback window, remove old version	No incidents logged

Treat the gates as hard stops, not suggestions — if a metric crosses a trigger threshold, pause or roll back and investigate before retrying.

Variants

• Kubernetes rolling update checklist: readiness probes, maxSurge, maxUnavailable, and pod disruption budgets.
• Blue-green deployment checklist: traffic switch mechanics, database compatibility, and version retention.
• Canary deployment checklist: metric thresholds, progressive traffic weights, and automated rollback gates.
• Serverless deployment checklist: function versioning, alias routing, and API Gateway stage management.
• Database-heavy deployment checklist: schema compatibility, migration order, and tested rollback scripts.
• Mobile or client deployment checklist: staged rollout, forced-update handling, and API compatibility.

What Works

• Keep deployments small and frequent — a 20-line diff is easier to roll back than a 2,000-line one.
• Keep database changes readable by both the old and the new code paths.
• Use health checks that verify real dependencies, not just process liveness.
• Automate rollback on metric thresholds instead of relying on someone watching a dashboard.
• Monitor business metrics alongside technical ones; a deploy can pass every health check and still break checkout.
• Keep a known-good baseline artifact so rollback is a redeploy, not a rebuild.
• Practice rollbacks in staging or game days — an untested rollback plan is a hypothesis.
• Record deployment decisions and outcomes so the checklist improves after every incident.

Common Mistakes

• Treating an HTTP 200 health check as proof the service works.
• Shipping a destructive schema change in the same release as the code that depends on it.
• Pushing 100% of traffic before canary metrics have been validated.
• Starting a deploy with no tested rollback path.
• Watching only the error rate while p99 latency quietly triples.
• Cleaning up the old version before the rollback window closes.
• Deploying into peak traffic without headroom for doubled capacity during the rollout.
• Trusting DNS for instant traffic switching while clients cache the old record.

Troubleshooting

• Canary passes health checks but users see errors: the readiness probe isn’t covering the failing dependency. Add the real dependency to /health/ready and re-run the rollout in staging.
• Rollback completes but the outage continues: the schema change wasn’t backward compatible, so the old code crashes against the new schema. Restore compatibility first, then roll back — this is why expand-contract exists.
• Pods terminate mid-request: the preStop hook or endpoint removal is missing, so the pod dies before the load balancer stops routing to it. Add the sleep-then-SIGTERM hook and check ingress propagation delay.
• Canary metrics look fine but the 100% rollout fails: the canary sample was too small to surface the failure mode, usually a resource limit or cold-cache path that only appears at full traffic. Compare canary and full-fleet resource profiles.
• Feature flag rollback leaves inconsistent data: the flag toggled writes but not reads, or vice versa. Roll out flag changes so write and read paths switch together, and keep the flag for at least one release after full enablement.

Further Reading

• Kubernetes rolling update and probe documentation — the authoritative reference for maxSurge, maxUnavailable, and probe semantics.
• Argo Rollouts documentation — canary steps, AnalysisTemplate, and metric providers.
• Blue-Green Deployment Guide — the full guide when your rollout needs a parallel environment.
• Postmortems from public incidents — real failure modes worth adding to your own checklist.