Zero-Downtime Deployment Checklist

Overview

Zero-downtime deployments update production services without interrupting users. This checklist helps teams verify that health checks, traffic routing, database migrations, and rollback plans are in place before and during a release.

When to Use

• For alternatives, see Blue-Green Deployment.

• Releasing a new version of a user-facing service.

• Deploying schema or data migrations that affect multiple instances.

• Changing infrastructure that could impact availability.

• Introducing a new rollout strategy like canary or blue-green.

• Preparing for a high-traffic event where stability matters most.

Prerequisites

• A deployment pipeline with automated build, test, and publish stages.
• Health check endpoints that represent real application readiness.
• Load balancer, ingress, or traffic controller that supports gradual rollout.
• Database migration strategy that is backward compatible.
• Rollback plan with known good artifact and data state.
• Monitoring and alerting for error rate, latency, and business metrics.
• A communication plan for stakeholders and customers.

Solution

Checklist

1. Pre-Deployment Readiness

• Deployment change is approved and documented.
• Code is merged and artifact is built and tagged.
• Automated unit, integration, and contract tests pass.
• Database migrations are reviewed for backward compatibility.
• Feature flags are configured for safe enablement.
• Capacity and scaling limits are sufficient for expected traffic.
• Monitoring dashboards and alerts are active.
• On-call rotation is aware of the deployment window.
• Rollback steps are documented and tested in a non-production environment.
• Customer-facing communication is prepared if needed.

2. Health Check Configuration

Check	Endpoint	Success Criteria	Failure Action
Liveness	/health/live	HTTP 200	Restart container
Readiness	/health/ready	HTTP 200 and dependencies up	Stop traffic routing
Startup	/health/startup	HTTP 200	Delay rollout
Dependency	/health/deps	Database, cache, queue reachable	Alert and halt
Business	/health/business	Critical flow returns expected value	Page on-call

3. Rollout Strategy Selection

Strategy	Use Case	Risk Level	Rollback Speed
Rolling update	Stateless services, low risk	Low	Medium (terminate new pods)
Blue-green	Stateful sessions, predictable releases	Medium	Fast (switch traffic back)
Canary	High risk, measurable metrics	Medium	Fast (drain canary)
Feature flag	Gradual user exposure	Low	Instant (toggle off)
A/B deployment	Validate user behavior	Medium	Fast (re-route traffic)

4. Deployment Execution Steps

Step	Action	Verification
1	Deploy to staging and run smoke tests	Staging tests pass
2	Deploy canary or small subset	Health checks pass, error rate stable
3	Monitor key metrics for canary duration	Latency, error rate, business metrics within baseline
4	Increase traffic percentage gradually	Each stage passes health and metric checks
5	Complete rollout to 100%	All instances healthy and serving traffic
6	Validate production endpoints	Smoke tests and critical user flows pass
7	Keep old version available for rollback	Retain for defined rollback window
8	Confirm rollback window has passed	Remove old version or update artifact baseline

5. Database Migration Safety

• Migrations are additive and backward compatible with the previous version.
• Old code can read new schema without errors.
• New code can read old schema if a rollback is needed.
• Indexes are created concurrently where supported.
• Large migrations are split into smaller batches.
• Data backfill or migration jobs are idempotent.
• Rollback script or compensating operation is available.
• Database changes are tested in staging with production-like data.

6. Rollback Triggers

Trigger	Threshold	Action
Error rate spike	> 0.5% for 2 minutes	Pause rollout and investigate
Latency increase	p99 > baseline + 30% for 5 minutes	Roll back traffic
Business metric drop	Conversion rate drops > 5%	Roll back immediately
Health check failure	> 10% failing	Roll back immediately
Critical alert	Any P1 incident	Roll back and page on-call
Canary timeout	Canary stage exceeds duration without passing	Roll back canary

7. Post-Deployment Validation

• Application logs show no unexpected errors.
• Error rate and latency are within baseline.
• Business metrics are stable or improving.
• All feature flags are in the intended state.
• Old resources are cleaned up after the rollback window.
• Deployment summary is shared with the team.
• Any issues are logged in the issue tracker with owners.

Explanation

Zero-downtime deployments rely on three things: safe rollout mechanics, reliable health signals, and fast rollback. A checklist ensures that each release considers traffic routing, data compatibility, and observability before any user is exposed. Combining this discipline with automation reduces the risk of production incidents and improves release confidence.

Kubernetes Rolling Update Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v2.3.1
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 15 && kill -SIGTERM 1"]
      terminationGracePeriodSeconds: 60

Argo Rollouts Canary Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-gateway
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api-gateway
        - setWeight: 30
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api-gateway
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v2.3.1
          ports:
            - containerPort: 8080

Variants

• Kubernetes rolling update checklist: Focus on readiness probes, max surge, max unavailable, and pod disruption budgets.
• Blue-green deployment checklist: Focus on traffic switch, database compatibility, and version retention.
• Canary deployment checklist: Focus on metric thresholds, progressive traffic weights, and automated rollback gates.
• Serverless deployment checklist: Focus on function versioning, alias routing, and API Gateway stage management.
• Database-heavy deployment checklist: Focus on schema compatibility, migration order, and rollback scripts.
• Mobile or client deployment checklist: Focus on staged rollout, forced update handling, and API compatibility.

What Works

• Keep deployments small and frequent to reduce risk.
• Make database changes backward compatible with both old and new code.
• Use health checks that verify real dependencies, not just process liveness.
• Automate rollback based on metrics, not just manual decision gates.
• Monitor business metrics, not just technical metrics.
• Maintain a known-good baseline artifact for fast rollback.
• Practice rollbacks in staging or during game days.
• Document deployment decisions and outcomes for future reviews.

Common Mistakes

• Skipping health checks or using trivial HTTP 200 checks.
• Deploying database changes that are not backward compatible.
• Rolling out 100% traffic before validating metrics.
• Not having a rollback plan before starting the deployment.
• Ignoring increased latency in favor of error rate alone.
• Cleaning up old versions too early.
• Deploying during peak traffic without traffic capacity planning.

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the deployment and zero-downtime guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply zero-downtime deployment checklist when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Common Production Pitfalls

• Leaving required fields blank or using vague one-word answers.
• Filling the document once and never updating it after scope or decisions change.
• Storing the document where the team does not look during incidents or reviews.
• Not assigning an owner, due date, or review cadence.
• Copying boilerplate without removing sections that do not apply.
• Skipping version control, which makes rollback and accountability impossible.
• Failing to link the document to related decisions or follow-up actions.
• Avoiding quarterly reviews that would retire stale or unused sections.