Blue-Green Deployment Recipe

Overview

Blue-green deployment is a release strategy that maintains two identical production environments — one active (blue) and one idle (green). New versions deploy to the idle environment, get smoke-tested, and then traffic switches instantly. If problems arise, rollback is a single traffic switch away, taking seconds instead of hours.

This recipe covers concrete implementations — Nginx config switching, Kubernetes service selectors, and Route 53 weighted routing. For the strategic trade-offs against rolling and canary releases, see the deployment strategies guide; for the abstract pattern, see the blue-green deployment pattern.

When to Use

Use this resource when:

• Downtime during deployments is unacceptable (SLA > 99.9%). See Health Check Endpoint for readiness verification.
• You need instant rollback capability without redeploying. See Feature Flags for instant toggles.
• Running database migrations that must be backward-compatible. See Docker Compose Local Dev for local migration testing.
• Validating new releases against real production traffic via canary routing. See Istio Canary Deployment for progressive traffic shifting.

Solution

Nginx Traffic Switch (Bash)

#!/bin/bash
# Switch traffic from blue to green

BLUE_IP="10.0.1.10"
GREEN_IP="10.0.1.11"
NGINX_CONF="/etc/nginx/sites-enabled/app"

# Update upstream to point to green
sed -i "s/server $BLUE_IP:8080/server $GREEN_IP:8080/" $NGINX_CONF
nginx -s reload

# Health check on green
if ! curl -sf http://$GREEN_IP:8080/health; then
  # Rollback instantly
  sed -i "s/server $GREEN_IP:8080/server $BLUE_IP:8080/" $NGINX_CONF
  nginx -s reload
  echo "Rollback complete"
  exit 1
fi

Kubernetes Deployment with Service Switch

apiVersion: v1
kind: Service
metadata:
  name: app-active
spec:
  selector:
    version: blue  # Change to "green" to switch
  ports:
    - port: 80
      targetPort: 8080

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
      version: green
  template:
    metadata:
      labels:
        app: web
        version: green
    spec:
      containers:
        - name: app
          image: myapp:v2.0.0

AWS Route 53 Weighted Routing

aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "SetIdentifier": "green",
        "Weight": 100,
        "TTL": 60,
        "ResourceRecords": [{"Value": "1.2.3.4"}]
      }
    }]
  }'

Explanation

How it works:

1. Blue environment serves all production traffic
2. Green environment receives the new deployment
3. Smoke tests validate green (health checks, synthetic transactions)
4. Traffic switch routes all users to green
5. Blue remains warm as instant rollback target
6. Next deploy goes to blue, roles swap

Database compatibility:

• Migrations must be backward-compatible (add columns, never remove)
• Blue must tolerate green’s schema; green must tolerate blue’s schema
• Use feature flags to hide new columns from blue

Variants

Strategy	Downtime	Rollback Speed	Cost
Blue-Green	Zero	Instant (seconds)	2x infrastructure
Rolling	Zero	Slow (minutes)	1x + surge
Canary	Zero	Medium (minutes)	1x + small surge
Recreate	High	N/A	1x

What Works

• Keep blue warm for one deploy cycle: Don’t decommission until the next deployment succeeds
• Automate the switch: Manual DNS updates are error-prone; use CI/CD pipelines
• Monitor during switch: Watch error rates, latency, and business metrics for 5-10 minutes post-switch
• Use sticky sessions carefully: Switching mid-session can disrupt stateful connections
• Share state via external stores: Both environments must access the same databases, caches, and queues

Common Mistakes

1. Stateful blue/green: Sessions stored in-memory are lost during switches; use Redis or JWT
2. Different configs: Blue and green must have identical environment variables except the version
3. Forgetting to test rollback: A deployment that can’t roll back safely is not production-ready
4. Database schema conflicts: Breaking schema changes deployed before both apps are compatible
5. Leaving old environments running: Unused environments incur cloud costs; automate cleanup

Production Automation

CI/CD Pipeline with GitHub Actions

# .github/workflows/blue-green.yml
name: Blue-Green Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to green
        run: |
          kubectl apply -f k8s/green-deployment.yaml
          kubectl rollout status deployment/app-green --timeout=120s

      - name: Smoke test green
        run: |
          GREEN_IP=$(kubectl get svc app-green -o jsonpath='{.spec.clusterIP}')
          for i in $(seq 1 10); do
            if curl -sf "http://$GREEN_IP:8080/health"; then
              echo "Green healthy"
              break
            fi
            sleep 5
          done

      - name: Switch traffic to green
        run: |
          kubectl patch svc app-active -p '{"spec":{"selector":{"version":"green"}}}'

      - name: Verify production
        run: |
          sleep 30
          if ! curl -sf https://api.example.com/health; then
            kubectl patch svc app-active -p '{"spec":{"selector":{"version":"blue"}}}'
            echo "Rollback triggered"
            exit 1
          fi

      - name: Scale down blue
        run: kubectl scale deployment app-blue --replicas=0

Database Migration Strategy

import psycopg2

def expand_migrate(conn):
    """Phase 1: Expand — add new columns, keep old ones."""
    with conn.cursor() as cur:
        cur.execute("""
            ALTER TABLE users
            ADD COLUMN IF NOT EXISTS email_v2 VARCHAR(255);
        """)
        cur.execute("""
            UPDATE users SET email_v2 = email;
        """)
        conn.commit()

def contract_migrate(conn):
    """Phase 2: Contract — remove old columns after blue is decommissioned."""
    with conn.cursor() as cur:
        cur.execute("""
            ALTER TABLE users DROP COLUMN IF EXISTS email;
        """)
        conn.commit()

# Deploy sequence:
# 1. Run expand_migrate (both blue and green work)
# 2. Switch traffic to green
# 3. After validation, run contract_migrate (only green uses new schema)

Monitoring Script Post-Switch

#!/bin/bash
# monitor-post-switch.sh

DURATION=300  # 5 minutes
INTERVAL=10
END=$((SECONDS + DURATION))

while [ $SECONDS -lt $END ]; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.example.com/health)
  LATENCY=$(curl -s -o /dev/null -w "%{time_total}" https://api.example.com/health)
  ERROR_RATE=$(curl -s https://api.example.com/metrics | grep error_rate | awk '{print $2}')

  echo "$(date -Iseconds) status=$STATUS latency=${LATENCY}s error_rate=$ERROR_RATE"

  if [ "$STATUS" != "200" ] || (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
    echo "ALERT: Rolling back"
    kubectl patch svc app-active -p '{"spec":{"selector":{"version":"blue"}}}'
    exit 1
  fi

  sleep $INTERVAL
done

echo "Monitoring complete. Green is stable."

Cost Optimization

1. Scale blue down between deploys. Keep blue at 1 replica to save costs while maintaining rollback capability:

kubectl scale deployment app-blue --replicas=1

2. Use spot instances for the idle environment. Green runs briefly during deploy; run it on cheaper spot instances:

nodeSelector:
  node-role.kubernetes.io/spot: "true"

3. Cache the Docker image on all nodes. Pre-pull the new image to avoid slow startup during switch:

# Use a DaemonSet to pre-pull
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-pre-puller
spec:
  template:
    spec:
      initContainers:
      - name: pull
        image: myapp:v2.0.0
        command: ["true"]
      containers:
      - name: pause
        image: gcr.io/google_containers/pause

Further Reading

• Kubernetes Services — selector-based traffic switching
• Route 53 weighted routing — DNS-level blue-green
• Lambda aliases — serverless blue-green

See Also

• Blue-Green Deployment Guide — strategic trade-offs and rollout planning
• Blue-Green Deployment Pattern — the abstract pattern
• Graceful Shutdown — draining connections before the switch
• Istio Canary Deployment — progressive traffic shifting instead
• Runnable scripts — Nginx switch, K8s manifests, CI/CD workflow, and monitoring script