Blue-Green Deployment Guide

Overview

Blue-green deployment keeps two production-grade environments alive: blue serves all traffic while green waits idle. You deploy the new version to green, validate it against production conditions, flip traffic over, and keep blue warm as your rollback insurance. If anything breaks, you flip back — seconds, not a redeploy.

The concept is simple; the operational decisions are not. Which mechanism flips the traffic? How do you migrate the database so both versions can run? How long should blue stay warm? This guide covers those decisions at the strategy level — architecture, switching mechanisms, database compatibility, and rollback planning. For copy-paste implementations of an Nginx switch, a Kubernetes Service selector, or Route 53 weighted records, use the blue-green deployment recipe. For the abstract pattern, see the blue-green deployment pattern.

When to Use

Blue-green earns its doubled infrastructure cost when:

• Rollback speed is the top requirement: one selector flip or DNS change restores the old version in seconds — useful when a bad release costs real money per minute.
• Deploys are infrequent and consequential: weekly or monthly releases of a core service, not dozens of daily pushes.
• You can keep environments isolated but share state: same database, same cache, same secret store, different app versions.
• Validation needs production-real conditions: the green environment serves real traffic only after the switch, but smoke tests run against the real network, real dependencies, real data.

When NOT to Use

Skip blue-green when:

• You deploy many times a day — keeping a full parallel environment idle 95% of the time burns money for little benefit. Rolling or canary deploys fit high-frequency cadences better.
• Database changes are frequent and incompatible — expand/contract migrations add real overhead; with schema changes on every release, blue-green’s database tax outweighs its rollback win.
• Long-lived connections dominate — WebSockets, SSE streams, and in-flight batch jobs don’t survive a hard traffic switch. You need connection draining, which canary-style gradual cutovers handle more naturally.
• Traffic is tiny or massive — tiny services don’t justify the complexity; massive ones make duplicating infrastructure genuinely expensive.

Core Concepts

Four properties define a real blue-green setup:

1. Environment symmetry: blue and green run identical infrastructure — same instance types, same scaling rules, same network zone. “Green is a smaller copy” breaks the moment green takes full traffic.
2. A single switch point: traffic flips at one well-defined boundary — a load balancer listener rule, a DNS record, a Kubernetes Service selector. If the flip touches more than one place, you don’t have a switch, you have a sequence of partial failures.
3. Shared state, versioned code: both environments hit the same database, cache, and queue. The environments differ in code version only — never in data.
4. A warm rollback target: blue stays running (possibly scaled down) until the next release proves green. Decommissioning too early deletes your undo button.

Architecture

The switch point is the heart of the design. Everything else — the environment, the tests, the rollback — hangs off that single decision.

Choosing Your Switch Mechanism

Three mechanisms dominate, and they differ in propagation speed, blast radius, and what breaks when they’re wrong:

Mechanism	Cutover speed	Rollback	Limits
Load balancer rule	Instant	Instant	Single region, needs L7-capable LB
Kubernetes Service selector	Instant (in-cluster)	Instant	K8s-only; external traffic still resolves the same LB IP
DNS weighted records	TTL-bound (30s–5min)	TTL-bound	Client resolvers cache; some traffic pins to old env
Service mesh / ingress	Instant, weighted	Instant, weighted	Highest complexity

Choose LB or Service selector when the app sits behind infrastructure you control — instant flip, instant rollback, no caching surprises. Choose DNS only when the two environments live in different regions or providers with no shared LB — and accept that some clients ride the old environment until TTLs expire. DNS-based switching also means “instant” rollback is actually “whenever the TTL says”, which is a real contract difference.

A common hybrid: external DNS for region-level failover, internal Service selectors for the in-cluster traffic flip.

One more criterion most teams discover late: test the failure mode, not just the happy path. DNS switching fails “soft” — clients keep hitting the old record until caches expire, and there is no way to force them. LB switching fails “hard” — a bad rule can take all traffic down at once. If your environment can’t survive a hard failure during the flip, build the flip as a small staged sequence (5% → 50% → 100% on the LB’s weighted targets) even though it looks like extra work. Blue-green purists call that canary; pragmatists call it not getting paged at 3 a.m.

Step-by-Step Blue-Green Deployment

1. Prepare the Green Environment

Provision green to match blue exactly — same instance profile, same autoscaling limits, same IAM roles. Deploy the new version, then run the real gates:

• Readiness probes return 200 on /health for every replica
• Synthetic transactions hit the critical path (login, checkout, whatever makes money)
• Dependency checks pass: DB connectivity, cache warmup, queue consumers started

Do not shortcut this with “pods are Running.” Green that can’t serve is a false sense of safety.

2. Switch Traffic

Flip the switch point atomically:

# Kubernetes: one patch, instant
kubectl patch svc app-active -p '{"spec":{"selector":{"version":"green"}}}'

# AWS ALB: point the listener rule at the green target group
aws elbv2 modify-rule --rule-arn arn:aws:elasticloadbalancing:... \
  --actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:...green

If your switch procedure has more than one step, write it as a script and test the script itself — a two-step manual flip is two chances to leave traffic split.

3. Monitor Post-Switch

Watch for 5–10 minutes minimum, with real thresholds defined before the switch:

• HTTP 5xx rate — rollback trigger at >1%
• p95/p99 latency — rollback if it degrades beyond 1.5x blue’s baseline
• Business metrics — signups/minute, orders/minute, job throughput; a silent data bug shows here before it shows in errors

Automate the observation: a loop hitting /health and /metrics every 10 seconds is enough for most teams; wire it to auto-rollback if you trust your thresholds.

4. Rollback (If Needed)

Rollback is the same flip in reverse — no redeploy, no migration reversal:

kubectl patch svc app-active -p '{"spec":{"selector":{"version":"blue"}}}'

Two rules: rollback within the window where the database is still compatible with both versions (see below), and log every rollback with its trigger — repeated rollbacks on the same release mean the smoke tests are lying to you.

5. Decommission Blue

Keep blue warm through one release cycle — the time between this deploy and the next. Then scale it down (replicas → 0 or a single warm instance) and reuse it as green for the next release. The environments swap roles every deploy; you never destroy either one permanently.

Sizing the Green Environment

“Identical to blue” is the rule, but there is room to be smart about cost. Three questions decide the sizing:

How long does green stay at full capacity? If your deploy window is under an hour, full-size green costs almost nothing — the expense is the warm standby period after the switch. Most teams run green at full size only during validation + switch, then drop blue to minimum warmth within the hour.

Can you tolerate cold start? Java/JVM apps and large connection pools take minutes to warm. If your rollback plan is “flip selector → users wait 90 seconds while green warms,” that’s a degraded rollback. Keep at least one warm instance in the standby environment at all times.

What does warm actually cost? Rough numbers: a single t3.large on AWS is ~$50/month; a managed K8s node, ~$70–150/month. Compare that against your incident cost: if one 30-minute outage costs more than a year of standby, the warm replica is the cheapest insurance in your stack.

A practical budget allocation: full-size green for the deploy window, one warm blue replica until the next release, and scheduled environments (scale to zero outside business hours) for pre-production tiers.

Observability and Rollback Triggers

Define the rollback thresholds before you deploy, not while the incident is unfolding. The signals that matter, in order of reliability:

Signal	Threshold	Why it catches what others miss
HTTP 5xx rate	>1% sustained 60s	Direct user impact
p99 latency	>1.5x blue baseline	Saturation problems that don’t error
Error log rate	>2x baseline	Bugs that return 200 with wrong data
Business metric drop	>20% vs same-hour baseline	Silent failures (broken checkout button, dead webhook)

Wire them in layers: infrastructure metrics alert first (seconds), error logs second (tens of seconds), business metrics last (minutes) but with the highest signal-to-noise for data bugs. Automated rollback on the infra layer is safe; automated rollback on business metrics is not — a marketing email going out can look like a conversion drop. Keep a human on the last layer.

Log every switch and every rollback with timestamp, trigger, and version pair — after three deploys, that log tells you more about your release quality than any dashboard.

Database Considerations

This is where blue-green deployments actually live or die. Both app versions share one database, so the schema has to satisfy both at once. The pattern that works is expand/contract:

Expand first — add what’s new without touching what exists:

-- Green can read/write email_v2; blue still sees only email.
ALTER TABLE users ADD COLUMN email_v2 VARCHAR(255);
UPDATE users SET email_v2 = email WHERE email_v2 IS NULL;

Deploy and switch — both versions coexist on the compatible schema; blue keeps using email, green uses email_v2, and a trigger or app-level dual-write keeps them in sync during the transition window.

Contract last — remove the old shape only after blue is decommissioned and you’re certain no rollback will need it:

ALTER TABLE users DROP COLUMN email;
ALTER TABLE users RENAME COLUMN email_v2 TO email;

The contract step can wait weeks. The cardinal sin is ordering it wrong: a destructive migration before the switch removes your rollback path entirely — if green fails, blue can’t even start against the new schema.

Deploy Day Timeline

A release that goes smoothly looks uneventful because the work happened before the window. A realistic timeline for a service that matters:

T-24h: Freeze feature merges to the release branch. Run the expand migration and confirm both app versions still boot against it. Rehearse the rollback in staging — actually flip the selector back, not just “read the command.”

T-2h: Scale green to full capacity and deploy the candidate. Record blue’s baselines: 5xx rate, p99 latency, throughput on your top three endpoints. Notify on-call.

T-15m: Run smoke tests against green through the internal endpoint — not the public URL. Verify synthetic transactions, dependency checks, and queue consumers. Confirm the rollback command works against the current infrastructure (it drifts).

T+0: Flip the switch point. This is the only manual step that should exist — everything before it was automation, everything after it is observation.

T+5m to T+15m: Watch the thresholds from the observability section. No browsing dashboards for “anything weird” — you’re checking four predefined numbers.

T+30m: Declare the release stable or roll back. Beyond 30 minutes, a rollback becomes harder to justify (and the data written since the switch needs reconciliation).

T+24h to next release: Blue stays warm at minimum capacity. Schedule the contract migration and the blue decommission as the same ticket — they’re both “finishing the release.”

What Works

• Rehearse the rollback as part of the deploy, not as a separate drill. The first time you flip back should never be during an incident.
• Keep blue at minimum warmth between deploys: one replica maintains the rollback option for pennies; zero means the next deploy builds green from scratch anyway.
• Version-tagged everything: image tags, Service selectors, target groups — if the selector doesn’t encode the version, the switch can’t find it.
• A written go/no-go checklist: smoke tests pass, metrics baselined, rollback command tested, on-call knows it’s deploy day. Two minutes of checklist beats an hour of “why is checkout down.”
• Sticky-session awareness: if sessions live in-memory, the switch orphans them mid-flight. Externalize sessions (Redis, JWT) before adopting blue-green — not during.
• Document the switch procedure where on-call will find it: the runbook belongs next to the alert, not in a wiki nobody opens during an incident. A pinned comment in the deploy pipeline with the exact rollback command has saved more releases than any architecture diagram.

Common Mistakes

1. Splitting traffic by accident: a manual two-step flip, or DNS plus LB rules that disagree, leaves some users on blue and some on green — two production versions, undefined behavior.
2. Contracting too early: dropping columns the moment green is healthy means a rollback resurrects code that reads a schema that no longer exists.
3. Decommissioning blue at switch+0: the incident that matters often arrives hours later. Keep blue warm until the next release.
4. Unequal environments: green on smaller instances or older AMIs means “green is slow” tells you nothing about the release.
5. Shared state surprises: queues consumed twice, caches written by both versions with different serializers, cron jobs running in both environments. Decide which environment owns scheduled work explicitly.
6. Rollback as improvisation: if the rollback path wasn’t tested last week, assume it doesn’t work.
7. Health checks that lie: a /health endpoint returning 200 while dependencies are dead means green passes every gate and fails the moment traffic arrives. Make readiness verify real dependencies — DB connection, cache hit, queue subscription — not just “the process is up.”

Variants

Scenario	Recommended shape
Single app behind one LB	ALB/NLB target-group swap or ingress annotation
Kubernetes cluster	Service selector patch (see the recipe) + readiness gates
Multi-region or multi-cloud	DNS weighted records + per-region LB flip
Serverless	Lambda aliases / API Gateway stages — blue-green without instances
Database-heavy release	Expand/contract + feature flags hiding new reads/writes

Further Reading

• Kubernetes Services — selector-based traffic switching
• AWS ALB listeners and rules — target-group swap mechanics
• Route 53 weighted routing — DNS-level cutover
• Lambda aliases — serverless blue-green

See Also

• Blue-Green Deployment Recipe — copy-paste implementations: Nginx, K8s Service, Route 53, CI/CD workflow
• Blue-Green Deployment Pattern — the abstract pattern
• Deployment Strategies Guide — blue-green vs rolling vs canary
• Canary Deployment Guide — gradual traffic shifting
• Disaster Recovery Guide — when rollback is part of a bigger plan