Failover: Switch to Standby on Primary Failure Detection

Overview

Your primary node will fail eventually — a kernel panic, a crashed deploy, an AWS region having a bad day. The failover pattern answers the question of what happens next: a health checker notices the primary is down and redirects traffic to a standby that was already running. Done well, users see a brief blip; done badly, they see a 502 page until someone wakes up and fixes it.

The pattern is occasionally spelled “fallover” in older vendor docs, but “failover” is the standard term in AWS, Azure, Kubernetes, and PostgreSQL documentation, and it’s what people actually search for.

There are two main configurations. In active-passive, one node serves everything and the standby waits idle until it’s promoted. In active-active, both nodes serve traffic and a failure simply shrinks capacity: the surviving node absorbs the full load. Failover can also happen at four different layers: DNS (repoint a record to another IP), load balancer (stop sending requests to an unhealthy upstream), database (promote a read replica to writable primary), or application level (the client retries against a backup endpoint). Each layer trades switchover speed against complexity, and most production systems combine two or three of them.

When to Use

• Services with an availability target that a single node can’t meet (99.9%+ usually implies automatic failover)
• Stateful systems like databases where the primary can die mid-transaction and a replica must take over
• Multi-region deployments where an entire region can become unreachable
• Calls to third-party APIs that publish a documented backup endpoint or secondary region
• Disaster recovery plans that promise an RTO under a few minutes, which a manual restart almost never meets

When NOT to Use

• Single-instance apps where a few minutes of downtime costs less than running idle infrastructure
• Operations that must not complete twice: automatic promotion of an ambiguous primary can produce split-brain, which is worse than an outage. Financial ledgers often prefer manual failover with a human confirming the old primary is truly dead
• Stateless services behind a health-checking load balancer, since the LB already removes dead instances, so a second failover layer adds nothing
• When the standby would go stale anyway (no replication, no warm cache): the failover succeeds on paper and then collapses under load

Solution

Health-monitored failover (Python)

# failover/health_monitored.py — Active-passive failover with health checks
import time
import threading
import requests
from enum import Enum

class NodeStatus(Enum):
    HEALTHY = "healthy"
    UNHEALTHY = "unhealthy"
    UNKNOWN = "unknown"

class FailoverManager:
    """Monitors primary and standby nodes.
    Automatically fails over when primary becomes unhealthy."""

    def __init__(self, primary_url, standby_url, health_path="/health",
                 check_interval=5, failure_threshold=3, recovery_threshold=3):
        self.primary_url = primary_url
        self.standby_url = standby_url
        self.health_path = health_path
        self.check_interval = check_interval
        self.failure_threshold = failure_threshold
        self.recovery_threshold = recovery_threshold

        self._active_url = primary_url
        self._primary_failures = 0
        self._primary_successes = 0
        self._standby_failures = 0
        self._is_failover = False
        self._lock = threading.Lock()
        self._running = True

    @property
    def active_url(self):
        with self._lock:
            return self._active_url

    @property
    def is_failover(self):
        with self._lock:
            return self._is_failover

    def _check_health(self, url):
        """Check if a node is healthy."""
        try:
            resp = requests.get(f"{url}{self.health_path}", timeout=3)
            if resp.status_code == 200:
                return NodeStatus.HEALTHY
            return NodeStatus.UNHEALTHY
        except Exception:
            return NodeStatus.UNHEALTHY

    def _monitor_loop(self):
        """Continuously monitor the primary and fail over if needed."""
        while self._running:
            primary_status = self._check_health(self.primary_url)

            with self._lock:
                if not self._is_failover:
                    # Monitoring primary
                    if primary_status == NodeStatus.HEALTHY:
                        self._primary_failures = 0
                    else:
                        self._primary_failures += 1
                        if self._primary_failures >= self.failure_threshold:
                            print(f"Primary failed {self._primary_failures} times, "
                                  f"failing over to standby")
                            self._initiate_failover()
                else:
                    # In failover mode — check if primary recovered
                    if primary_status == NodeStatus.HEALTHY:
                        self._primary_successes += 1
                        if self._primary_successes >= self.recovery_threshold:
                            print(f"Primary recovered {self._primary_successes} times, "
                                  f"failing back")
                            self._initiate_failback()
                    else:
                        self._primary_successes = 0

            time.sleep(self.check_interval)

    def _initiate_failover(self):
        """Switch traffic from primary to standby."""
        standby_status = self._check_health(self.standby_url)
        if standby_status == NodeStatus.HEALTHY:
            self._active_url = self.standby_url
            self._is_failover = True
            self._primary_successes = 0
            print(f"Failover complete: now serving from {self.standby_url}")
        else:
            print(f"CRITICAL: Standby is also unhealthy! Cannot fail over.")

    def _initiate_failback(self):
        """Switch traffic back to primary."""
        self._active_url = self.primary_url
        self._is_failover = False
        self._primary_failures = 0
        print(f"Failback complete: now serving from {self.primary_url}")

    def start(self):
        """Start the monitoring thread."""
        t = threading.Thread(target=self._monitor_loop, daemon=True)
        t.start()

    def stop(self):
        self._running = False

    def request(self, method, path, **kwargs):
        """Make a request to the currently active node."""
        url = f"{self.active_url}{path}"
        return requests.request(method, url, **kwargs)

    def get_status(self):
        with self._lock:
            return {
                "active_url": self._active_url,
                "is_failover": self._is_failover,
                "primary_failures": self._primary_failures,
                "primary_successes": self._primary_successes,
            }


# Usage
failover = FailoverManager(
    primary_url="https://api-primary.example.com",
    standby_url="https://api-standby.example.com",
    health_path="/health",
    check_interval=5,
    failure_threshold=3,
    recovery_threshold=3
)
failover.start()

# All requests go to the active node (primary or standby)
response = failover.request("GET", "/api/products")

The important detail is the threshold logic: a single failed check doesn’t trigger a switch. Requiring three consecutive failures before acting, and three consecutive successes before failing back, is what separates a failover system from a flapping one.

Database failover with PostgreSQL (Python)

# failover/database.py — PostgreSQL failover with connection switching
import psycopg2
import time
import threading

class DatabaseFailover:
    """Manages database connections with automatic failover.
    Primary: read-write. Standby: read-only, promoted on failover."""

    def __init__(self, primary_config, standby_configs):
        self.primary_config = primary_config
        self.standby_configs = standby_configs
        self._active_config = primary_config
        self._is_failover = False
        self._lock = threading.Lock()
        self._connection = None

    def _create_connection(self, config):
        return psycopg2.connect(
            host=config["host"],
            port=config.get("port", 5432),
            database=config["database"],
            user=config["user"],
            password=config["password"],
            connect_timeout=5
        )

    def get_connection(self):
        """Get a connection to the active database."""
        with self._lock:
            if self._connection and not self._connection.closed:
                try:
                    # Test the connection
                    self._connection.cursor().execute("SELECT 1")
                    return self._connection
                except Exception:
                    self._connection = None

            # Try active config
            try:
                self._connection = self._create_connection(self._active_config)
                return self._connection
            except Exception as e:
                print(f"Active DB unavailable: {e}")
                self._initiate_failover()
                self._connection = self._create_connection(self._active_config)
                return self._connection

    def _initiate_failover(self):
        """Try each standby in order."""
        for i, standby in enumerate(self.standby_configs):
            try:
                conn = self._create_connection(standby)
                conn.close()
                self._active_config = standby
                self._is_failover = True
                print(f"Failed over to standby {i}: {standby['host']}")
                return
            except Exception:
                continue

        raise Exception("All databases are unavailable")

    def execute(self, query, params=None):
        """Execute a query on the active database."""
        conn = self.get_connection()
        cur = conn.cursor()
        cur.execute(query, params)
        result = cur.fetchall()
        conn.commit()
        return result

    @property
    def is_failover(self):
        with self._lock:
            return self._is_failover


# Usage
db = DatabaseFailover(
    primary_config={"host": "db-primary.internal", "database": "shop",
                    "user": "app", "password": "secret"},
    standby_configs=[
        {"host": "db-replica-1.internal", "database": "shop",
         "user": "app", "password": "secret"},
        {"host": "db-replica-2.internal", "database": "shop",
         "user": "app", "password": "secret"},
    ]
)

# Automatically fails over if primary is down
users = db.execute("SELECT * FROM users LIMIT 10")

In production you wouldn’t hand-roll this. PostgreSQL has pg_promote(), repmgr, or managed failover in RDS/Cloud SQL, and most drivers accept a list of hosts (target_session_attrs=read-write in libpq). The point of the example is the semantics: probe before trusting, try standbys in order, and fail loudly when nothing is left.

DNS-based failover (Python)

# failover/dns.py — DNS-based failover for multi-region
import dns.resolver
import time

class DNSFailover:
    """DNS-based failover: updates DNS records to point to standby.
    Slower propagation but works across regions."""

    def __init__(self, domain, primary_ip, standby_ip,
                 dns_server, ttl=60):
        self.domain = domain
        self.primary_ip = primary_ip
        self.standby_ip = standby_ip
        self.dns_server = dns_server
        self.ttl = ttl
        self._active_ip = primary_ip

    def check_and_failover(self, health_url):
        """Check primary health and update DNS if needed."""
        import requests
        try:
            resp = requests.get(health_url, timeout=5)
            if resp.status_code == 200:
                if self._active_ip != self.primary_ip:
                    self._update_dns(self.primary_ip)
                    self._active_ip = self.primary_ip
                    print(f"DNS failback to primary: {self.primary_ip}")
                return True
        except Exception:
            pass

        # Primary is down — fail over
        if self._active_ip == self.primary_ip:
            self._update_dns(self.standby_ip)
            self._active_ip = self.standby_ip
            print(f"DNS failover to standby: {self.standby_ip}")

        return False

    def _update_dns(self, ip):
        """Update DNS A record (implementation depends on DNS provider)."""
        # Example: AWS Route 53 API call
        # change_route53_record(self.domain, ip, self.ttl)
        print(f"Updating DNS: {self.domain} -> {ip} (TTL: {self.ttl}s)")

    def resolve_current(self):
        """Check what IP the domain currently resolves to."""
        resolver = dns.resolver.Resolver()
        answers = resolver.resolve(self.domain, "A")
        return [rdata.address for rdata in answers]

DNS failover is the only option that survives a whole-region outage, because it works above the infrastructure that failed. The catch is propagation: resolvers cache your record for the TTL, and some ISPs cache longer than they should. With a 60-second TTL, expect most clients to move within a couple of minutes, not the seconds a load balancer gives you.

Nginx upstream failover

# failover/nginx.conf — Nginx upstream with passive failover
upstream api_backend {
    # Primary server — receives all traffic when healthy
    server api-primary:8080 max_fails=3 fail_timeout=30s;

    # Standby server — receives traffic when primary fails
    server api-standby:8080 backup max_fails=3 fail_timeout=30s;

    # Health check settings
    keepalive 32;
    keepalive_timeout 60s;
}

server {
    listen 80;
    server_name api.example.com;

    # Active health checks (requires nginx plus)
    # health_check interval=5s fails=3 passes=2 uri=/health;

    location / {
        proxy_pass http://api_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_connect_timeout 5s;
        proxy_read_timeout 30s;
    }

    # Health endpoint for external monitoring
    location /health {
        access_log off;
        return 200 "healthy\n";
        add_header Content-Type text/plain;
    }
}

The backup directive is what makes the second server a standby rather than a load-balanced peer: Nginx only sends it traffic once the primary exceeds max_fails within fail_timeout. Open-source Nginx only does passive checks — it learns a server is dead from real requests failing, which means the first few users absorb the errors. Paid Nginx Plus, HAProxy, Envoy, and cloud load balancers all do active probing and detect failure before users do.

Kubernetes multi-cluster failover

# failover/k8s-failover.yaml — Kubernetes service with failover
apiVersion: v1
kind: Service
metadata:
  name: api-service
  annotations:
    # External-dns annotation for DNS-based failover
    external-dns.alpha.kubernetes.io/hostname: api.example.com
spec:
  type: LoadBalancer
  selector:
    app: api-server
  ports:
    - port: 80
      targetPort: 8080
---
# PodDisruptionBudget ensures minimum availability
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api-server
---
# Multi-cluster failover with Global Load Balancer (e.g., AWS Global Accelerator)
# Primary cluster: us-east-1
# Standby cluster: eu-west-1
# Traffic routes to primary; on failure, routes to standby

Inside one cluster, kubelet already restarts dead pods and the Service keeps endpoints fresh, which is failover you get for free. The harder problem this config hints at is cluster-level failover: two clusters in different regions behind a global anycast load balancer (AWS Global Accelerator, Cloudflare Load Balancing, Azure Front Door) that health-checks both and shifts traffic when one goes dark.

JavaScript failover client

// failover/client.js — Client-side failover for API calls
class FailoverClient {
    constructor(endpoints, options = {}) {
        this.endpoints = endpoints;  // ["https://api1.com", "https://api2.com"]
        this.activeIndex = 0;
        this.healthPath = options.healthPath || "/health";
        this.checkInterval = options.checkInterval || 10000;
        this.failureThreshold = options.failureThreshold || 3;
        this.failures = 0;
        this.isChecking = false;
    }

    get activeEndpoint() {
        return this.endpoints[this.activeIndex];
    }

    async checkHealth() {
        try {
            const response = await fetch(
                `${this.activeEndpoint}${this.healthPath}`,
                { signal: AbortSignal.timeout(3000) }
            );
            if (response.ok) {
                this.failures = 0;
                return true;
            }
        } catch (error) {
            // Health check failed
        }

        this.failures++;
        if (this.failures >= this.failureThreshold) {
            this.failover();
        }
        return false;
    }

    failover() {
        const nextIndex = (this.activeIndex + 1) % this.endpoints.length;
        if (nextIndex !== this.activeIndex) {
            console.log(`Failing over from ${this.endpoints[this.activeIndex]} ` +
                        `to ${this.endpoints[nextIndex]}`);
            this.activeIndex = nextIndex;
            this.failures = 0;
        }
    }

    async request(method, path, options = {}) {
        const url = `${this.activeEndpoint}${path}`;
        try {
            const response = await fetch(url, {
                method,
                ...options,
                signal: AbortSignal.timeout(10000)
            });
            if (response.status >= 500) {
                this.failures++;
                if (this.failures >= this.failureThreshold) {
                    this.failover();
                    // Retry on the new endpoint
                    return this.request(method, path, options);
                }
                throw new Error(`HTTP ${response.status}`);
            }
            this.failures = 0;
            return response;
        } catch (error) {
            this.failures++;
            if (this.failures >= this.failureThreshold) {
                this.failover();
                return this.request(method, path, options);
            }
            throw error;
        }
    }

    startHealthChecks() {
        this.isChecking = true;
        const check = async () => {
            if (!this.isChecking) return;
            await this.checkHealth();
            setTimeout(check, this.checkInterval);
        };
        check();
    }

    stopHealthChecks() {
        this.isChecking = false;
    }
}

// Usage
const client = new FailoverClient(
    ["https://api-primary.example.com", "https://api-standby.example.com"],
    { failureThreshold: 3, checkInterval: 10000 }
);
client.startHealthChecks();

const response = await client.request("GET", "/api/products");
const data = await response.json();

Client-side failover is the last resort: it works when nothing between you and the API can do it for you, but every client keeps its own view of which endpoint is alive, so a fleet of clients can disagree during a partial outage. Prefer it for a small number of known endpoints, not as a general load-balancing strategy. Pair it with Retry with Jitter so the retries don’t hammer a node that’s still recovering.

How It Works

Every failover implementation, at any layer, runs the same four-step loop:

1. Detect. A health check hits an endpoint like /health every few seconds. A good health endpoint checks real dependencies (database connectivity, disk, queue lag), not just “is the process up”. A node that returns 200 while its database is unreachable will keep the traffic and fail every request.

2. Decide. One failure is noise; several in a row is a signal. The failure threshold is the main tuning knob: too low and you flap on a dropped packet, too high and users wait while you count. Three consecutive failures with a 5-second interval, roughly 15 seconds to detect, is a sane default.

3. Switch. Redirect traffic to the standby. What “redirect” means depends on the layer, and the layer determines how fast users feel the switch:

Layer	Mechanism	Typical switch time	Main trade-off
Application client	Retry against backup endpoint list	Sub-second	Every client keeps its own state
Load balancer	Remove unhealthy upstream (backup, health checks)	Seconds	Only covers what sits behind that LB
DNS	Update A record to standby IP	1–5 min (TTL-bound)	Slowest, but survives regional outages
Database	Promote replica to writable primary	30–120 s	Must verify replication lag first

4. Recover (failback). When the primary comes back, the checker needs to see it healthy several times before trusting it again, because a node that just rebooted often passes one check and dies under real load. Some teams skip automatic failback entirely: they switch once, investigate the primary, and fail back manually during a quiet window. That’s a defensible choice, not laziness.

The failure modes worth losing sleep over:

• Split-brain: a network partition makes both nodes believe they’re primary, and both accept writes. Fencing (STONITH), a quorum, or a leader-election system like etcd prevents it.
• Failover flap: thresholds too tight cause the system to oscillate primary → standby → primary every few seconds. Hysteresis — different thresholds for failing over and failing back — fixes it.
• Cold standby: the standby promotes cleanly but has no warm cache, no open connections, and falls over under sudden full load. Send it 1–5% of traffic permanently so it stays warm.
• Data loss window: with async replication, the last seconds of writes on the dead primary never made it to the standby. That gap is your real RPO, and no amount of health-check tuning shrinks it. Only synchronous replication does, at a latency cost.

Variants

Active-active with load balancing

# failover/active_active.py — Both nodes serve traffic simultaneously
import random
import requests

class ActiveActiveManager:
    """Both primary and standby serve traffic.
    If one fails, the other absorbs all traffic."""

    def __init__(self, endpoints, health_path="/health"):
        self.endpoints = {url: {"healthy": True, "failures": 0}
                          for url in endpoints}
        self.health_path = health_path

    def get_healthy_endpoints(self):
        return [url for url, info in self.endpoints.items()
                if info["healthy"]]

    def request(self, method, path, **kwargs):
        healthy = self.get_healthy_endpoints()
        if not healthy:
            raise Exception("All endpoints are unhealthy")

        # Random load balancing among healthy endpoints
        url = random.choice(healthy)
        try:
            resp = requests.request(method, f"{url}{path}", timeout=10, **kwargs)
            return resp
        except Exception:
            self.endpoints[url]["failures"] += 1
            if self.endpoints[url]["failures"] >= 3:
                self.endpoints[url]["healthy"] = False
                print(f"Marked {url} as unhealthy")
            # Retry on another healthy endpoint
            return self.request(method, path, **kwargs)

Active-active eliminates the idle-standby cost (every node you pay for does real work), and a “failover” is just a node dropping out of the rotation. The price is capacity planning: each node must handle 100% of traffic alone, so a two-node active-active pair really runs at 40% utilization in the steady state. For stateful services, active-active also forces you to solve multi-writer conflicts, which is why it’s common for stateless APIs and rare for databases.

Cascading failover (multi-tier)

# failover/cascading.py — Multi-tier failover: primary -> standby -> tertiary
import requests

class CascadingFailover:
    """Tries endpoints in order: primary, then standby, then tertiary.
    Each tier is tried only if the previous one fails."""

    def __init__(self, tiers):
        """tiers: [{"name": "primary", "url": "...", "timeout": 5}, ...]"""
        self.tiers = tiers
        self._active_tier = 0

    @property
    def active_url(self):
        return self.tiers[self._active_tier]["url"]

    def request(self, method, path, **kwargs):
        for i, tier in enumerate(self.tiers):
            try:
                url = f"{tier['url']}{path}"
                resp = requests.request(method, url, timeout=tier["timeout"], **kwargs)
                if resp.status_code < 500:
                    if i != self._active_tier:
                        print(f"Failover: tier {self._active_tier} -> {i} ({tier['name']})")
                        self._active_tier = i
                    return resp
            except Exception:
                continue

        raise Exception("All tiers exhausted")

Cascading failover adds a third tier, useful when the standby itself might be degraded, or when you want a cheap last-resort option (a static fallback page, a read-only replica, a reduced-functionality mode) instead of a full error. Each extra tier multiplies the states you need to test, so stop at three.

Best Practices

• Automate detection end to end. A failover that requires someone to notice a dashboard alert and press a button is a manual restart with extra steps.
• Require several consecutive failures before switching. Three failures at a 5-second interval detects a real outage in ~15 seconds without flapping on a single dropped packet.
• Test failover regularly. Run game days where you actually kill the primary, under realistic traffic, not on an idle staging box.
• Alert on every failover event. A switch means something broke, and silence is how a degraded standby runs for weeks until it becomes the primary and dies too.
• Keep the standby warm: send it a small share of traffic so caches, connections, and TLS sessions are already established. A cold standby is a failover that works on paper.
• Use short DNS TTLs (60 seconds or less) on any record you expect to move during an incident.
• Verify replication lag before promoting a database standby; promoting a replica that’s 30 seconds behind silently discards 30 seconds of writes.
• Plan the failback before you need it, including who approves it and during which traffic window it runs. For graceful restarts on the returning node, see Graceful Shutdown: Drain In-Flight Requests Before Exit.

Common Mistakes

• Health check that only checks “process is up”: the node returns 200 while its database connection is dead, so traffic keeps flowing to a zombie. Check real dependencies.
• Failure threshold of one: a single dropped packet triggers a switch, the system oscillates, and you get a worse outage than the one you were preventing.
• Standby that never gets exercised: it fails over for the first time in months during a real incident, and turns out to be misconfigured. Promote it deliberately every few weeks.
• No failback plan: the “temporary” standby becomes the de facto primary, undocumented and unmonitored.
• Split-brain writes: automatic promotion without fencing lets an isolated-but-alive primary keep accepting writes alongside the new one. If you can’t fence, prefer manual promotion.
• Forgetting the dependency layer: the app fails over cleanly but its Redis, queue, or file store still points at the dead region. Failover has to cover the whole request path, not just the front door.

See Also

• NGINX upstream module documentation: backup, max_fails, and fail_timeout semantics for passive failover.
• Kubernetes PodDisruptionBudget: keep a minimum number of replicas alive during voluntary disruptions.
• Amazon Route 53 DNS failover: health-checked DNS records for multi-region failover.
• PostgreSQL warm standby / log shipping: how replicas are promoted to primary.
• Circuit Breaker Half-Open: Probe Recovery Before Reopening: the complementary pattern for calling a struggling dependency instead of replacing it.
• Retry with Jitter: Exponential Backoff for Transient Failures: what clients should do in the seconds before failover completes.

Companion code: failover examples: runnable versions of the snippets on this page.