StackPractices
intermediate By Mathias Paulenko

Leader Election Pattern

Coordinate a single active instance among multiple distributed nodes to avoid conflicts and split-brain scenarios.

Overview

The Leader Election Pattern makes sure that exactly one instance in a distributed system acts as the active leader at any time. The rest stay as followers and take over only when the leader stops renewing its claim. Without it, two nodes that each believe they’re in charge produce split-brain: duplicated jobs, conflicting writes, and corrupted shared state.

The pattern shows up wherever a cluster needs a single coordinator: kube-scheduler and kube-controller-manager elect a leader through Kubernetes Leases, Kafka picks one controller per cluster, and schedulers like Quartz or cron runners use it so a job fires once instead of N times. Under the hood it’s a specialization of the distributed lock pattern: the “lock” is a lease with a TTL that the winner must keep renewing.

When to Use

Reach for this pattern when:

  • Several replicas could run the same scheduled job, migration, or report, but the job must execute exactly once per interval.
  • A service needs a single writer for a shared resource (a queue, a config namespace, a shard assignment).
  • You’re building a cluster manager or control plane where a single coordinator simplifies decisions.
  • You can accept a failover gap measured in seconds while followers detect the dead leader and elect a new one.

Don’t use it when:

  • A single replica behind a health-checked load balancer already gives you the uptime you need, and election adds failure modes you don’t have yet.
  • The work tolerates duplication through idempotency or at-least-once semantics; a plain queue consumer group is simpler.
  • All nodes must keep working in parallel — that’s a work-distribution problem, not a coordination one.

Solution

Every correct implementation reduces to the same contract: candidates race to atomically write their identity into a shared key with an expiry (the lease), the winner refreshes it on a heartbeat, and the store returns a fencing token — a strictly increasing number — that downstream services check to reject stale leaders.

# Leader election over an atomic store with CAS and fencing tokens.
# Works the same on etcd Txn, ZooKeeper, or SQL `UPDATE ... WHERE`.
import threading
import time
import uuid


class AtomicStore:
    """Minimal stand-in for etcd/ZooKeeper: atomic ops only."""

    def __init__(self):
        self._data, self._fence, self._lock = {}, 0, threading.Lock()

    def get(self, key):
        return self._data.get(key)

    def cas(self, key, expected, value):
        """Write only if current == expected. Returns a fencing token."""
        with self._lock:
            if self._data.get(key) != expected:
                return None
            self._fence += 1
            value["fence"] = self._fence
            self._data[key] = value
            return self._fence


class LeaderElection:
    def __init__(self, store, lease_seconds=10):
        self.store = store
        self.lease_seconds = lease_seconds
        self.node_id = uuid.uuid4().hex[:8]

    def campaign(self):
        """Return a fencing token once this node wins the lease."""
        now = time.monotonic()  # elapsed time, immune to wall-clock jumps
        current = self.store.get("leader")
        if current and current["expires"] > now:
            return None  # someone else holds a live lease
        return self.store.cas(
            "leader", current,
            {"node": self.node_id, "expires": now + self.lease_seconds},
        )

    def renew(self, token):
        now = time.monotonic()
        current = self.store.get("leader")
        if not current or current["fence"] != token:
            return False  # lost leadership
        current["expires"] = now + self.lease_seconds
        return True

On Kubernetes you don’t write this yourself: the API server stores Lease objects and client-go’s leaderelection package runs the campaign:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: app-leader
  namespace: default
spec:
  holderIdentity: pod-1
  leaseDurationSeconds: 15
kubectl apply -f lease.yaml
kubectl get lease app-leader -o yaml   # inspect holder + renewTime

Explanation

The lifecycle has four moving parts:

  1. Campaign: each candidate tries one atomic compare-and-swap on the lease key. Only one write can win — that’s what makes the election safe even under a partition, as long as the store itself is consistent.
  2. Heartbeat: the winner refreshes the lease well before its TTL (typically at TTL/2 or TTL/3). Missing a single renewal isn’t fatal; missing all of them until expiry is.
  3. Expiry: when the renewals stop (crash, GC pause, network partition), the lease simply times out. Nothing has to “detect” the failure; time does it.
  4. Failover: a follower wins the next campaign and becomes leader with a higher fencing token.
Lease lifecycle — candidates race to grab the lease key; the winner heartbeats until it stops, the lease expires at its TTL, and a follower wins the next round.

Two details separate a toy implementation from a safe one:

Fencing tokens. A leader that loses its lease doesn’t find out instantly — it can be paused by a GC stop or partitioned away while still believing it’s in charge. If it then writes to a shared resource, you get two writers. The store returns a monotonic token on every successful election; the leader attaches it to each write, and downstream rejects anything with a token lower than the highest it’s seen. Raft calls this the term; etcd exposes it as the key’s revision. Skipping fencing is the most common way leader election code silently corrupts data.

Clock assumptions. Lease logic should run on elapsed time (time.monotonic, System.nanoTime), not wall-clock time — an NTP correction can otherwise “expire” every lease at once or extend one forever. The store’s own TTL (etcd lease TTL, K8s leaseDurationSeconds, Redis PX) is even better because expiry is decided by one clock, not by every node’s.

Picking the lease duration is the main operational trade-off: a short TTL (5-15s) fails over fast but flaps under load spikes or GC pauses; a long TTL is stable but leaves the cluster headless for longer. Most systems land at 15-30s with renewals at TTL/3.

Variants

VariantCoordination StoreTrade-off
Database leasePostgreSQL, MySQLSimple — one UPDATE ... WHERE expires < now() — but ties failover to DB availability
Distributed lockRedis SET NX PXFast and cheap, but vulnerable to clock drift and needs fencing checks done client-side
Consensus algorithmetcd, ZooKeeperLinearizable writes and watches built in; you run (or pay for) a consensus cluster
Kubernetes LeaseK8s API serverNative for workloads already on K8s; leaderelection package handles the loop

Choose the store your cluster already trusts. If you run etcd, use its concurrency.Election primitive (documented in the etcd concurrency API) — it orders candidates by create-revision, so there’s a clean succession line instead of a thundering herd on expiry. On Kubernetes, the Lease object plus client-go’s leaderelection is the paved road. Redis works for soft coordination where a rare double-leader is tolerable, but read the Redlock debate before betting correctness on it — Martin Kleppmann’s analysis of fencing and clock assumptions applies directly to election.

A note on scope: election is per responsibility, not per cluster. Electing one leader per partition, shard, or job type scales coordination out and is how Kafka (one controller plus partition leaders) and Vitess (per-shard primaries) stay available under partial failure.

What Works

  • Keep the lease short and renew it well before expiry: TTL/3 renewals absorb a missed beat without triggering failover.
  • Issue a fencing token per term and verify it on every downstream write; without it, a partitioned ex-leader can still corrupt state.
  • Step down cleanly on shutdown: resign the lease or let it expire instead of holding followers hostage for a full TTL.
  • Watch the lease key from followers (etcd Watch, K8s watch, ZooKeeper exists watcher) instead of polling — you get failover latency without the query load.
  • Keep leader duties idempotent where possible, so a duplicated action during failover is a retry, not a corruption event.
  • Log every campaign, acquisition, and loss with the fencing token; leadership flapping is invisible without it.

Common Mistakes

  • No fencing check on writes: the classic failure is an old leader that wakes from a GC pause and writes over the new leader’s work. The lease alone can’t prevent this; only the token can.
  • Lease longer than the outage you fear: a 60s TTL means up to a minute of headless cluster on every leader crash.
  • Wall-clock comparisons: datetime.utcnow() across nodes invites split-brain on clock skew; use the store’s TTL or elapsed-time clocks.
  • Non-atomic check-then-set: reading the key and writing it in two calls lets two candidates both “win”. Election must be a single CAS/transaction.
  • Renewing in the same thread that does the work: if leader work blocks (long GC, synchronous I/O), the lease expires mid-term. Renew on its own scheduler.
  • Treating election as consensus: electing a leader coordinates one decision; it doesn’t make followers agree on state. If you need that, you need Raft or Paxos — see the concurrency patterns guide for how they compose.

Advanced Solutions

Leader election with etcd’s transaction API

The etcd3 Python client is synchronous; the safe primitive is client.transaction: one atomic compare-then-put guarded by a lease, not a check-then-set:

import time
import etcd3

client = etcd3.client(host="localhost", port=2379)
ELECTION_KEY = "/app/leader"
NODE_ID = "node-42"

# The lease dies automatically if this node stops refreshing it.
lease = client.lease(ttl=10)

# Single atomic op: write our ID only if the key doesn't exist yet
# (version == 0). The fencing equivalent is the key's mod_revision.
won = client.transaction(
    compare=[client.transactions.version(ELECTION_KEY) == 0],
    success=[client.transactions.put(ELECTION_KEY, NODE_ID, lease=lease)],
    failure=[],
)

if won:
    try:
        while True:
            lease.refresh()          # keep the lease alive
            time.sleep(3)            # renew well before the TTL
    finally:
        lease.revoke()               # step down cleanly on exit
else:
    # Follower path: watch the key; when it's deleted or re-created,
    # retry the transaction above.
    events, cancel = client.watch(ELECTION_KEY)

For stronger ordering (queued succession instead of a race on expiry), etcd exposes concurrency.Election, which assigns each candidate a key ordered by create-revision — the etcd equivalent of ZooKeeper’s sequential ephemeral nodes.

Kubernetes leader election with client-go

Inside a cluster, use the Lease resource and client-go’s update loop. The holderIdentity field is the claim; renewTime drives failover:

package main

import (
	"context"
	"fmt"
	"time"

	coordinationv1 "k8s.io/api/coordination/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

type LeaseLeaderElection struct {
	clientset      *kubernetes.Clientset
	leaseName      string
	leaseNamespace string
	holderIdentity string
	leaseDuration  time.Duration
}

func NewLeaseLeaderElection(leaseName, namespace, holderIdentity string, duration time.Duration) (*LeaseLeaderElection, error) {
	config, err := rest.InClusterConfig()
	if err != nil {
		return nil, err
	}

	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		return nil, err
	}

	return &LeaseLeaderElection{
		clientset:      clientset,
		leaseName:      leaseName,
		leaseNamespace: namespace,
		holderIdentity: holderIdentity,
		leaseDuration:  duration,
	}, nil
}

func (l *LeaseLeaderElection) Acquire(ctx context.Context) error {
	lease := &coordinationv1.Lease{
		ObjectMeta: metav1.ObjectMeta{
			Name:      l.leaseName,
			Namespace: l.leaseNamespace,
		},
		Spec: coordinationv1.LeaseSpec{
			HolderIdentity:       &l.holderIdentity,
			LeaseDurationSeconds: pointerToInt32(int32(l.leaseDuration.Seconds())),
			AcquireTime:          &metav1.Time{Time: time.Now()},
			RenewTime:            &metav1.Time{Time: time.Now()},
		},
	}

	for {
		current, err := l.clientset.CoordinationV1().Leases(l.leaseNamespace).Get(ctx, l.leaseName, metav1.GetOptions{})
		if err != nil {
			// Lease doesn't exist, try to create
			_, err = l.clientset.CoordinationV1().Leases(l.leaseNamespace).Create(ctx, lease, metav1.CreateOptions{})
			if err == nil {
				return nil // Acquired leadership
			}
			time.Sleep(time.Second)
			continue
		}

		// Check if lease is expired or we are the holder
		if current.Spec.HolderIdentity == nil || *current.Spec.HolderIdentity == l.holderIdentity {
			// Renew lease
			current.Spec.HolderIdentity = &l.holderIdentity
			current.Spec.RenewTime = &metav1.Time{Time: time.Now()}
			_, err = l.clientset.CoordinationV1().Leases(l.leaseNamespace).Update(ctx, current, metav1.UpdateOptions{})
			if err == nil {
				return nil // Leadership maintained
			}
		}

		time.Sleep(time.Second)
	}
}

func (l *LeaseLeaderElection) Renew(ctx context.Context) error {
	ticker := time.NewTicker(l.leaseDuration / 2)
	defer ticker.Stop()

	for {
		select {
		case <-ticker.C:
			lease, err := l.clientset.CoordinationV1().Leases(l.leaseNamespace).Get(ctx, l.leaseName, metav1.GetOptions{})
			if err != nil {
				return fmt.Errorf("failed to get lease: %w", err)
			}

			if lease.Spec.HolderIdentity == nil || *lease.Spec.HolderIdentity != l.holderIdentity {
				return fmt.Errorf("lost leadership")
			}

			lease.Spec.RenewTime = &metav1.Time{Time: time.Now()}
			_, err = l.clientset.CoordinationV1().Leases(l.leaseNamespace).Update(ctx, lease, metav1.UpdateOptions{})
			if err != nil {
				return fmt.Errorf("failed to renew lease: %w", err)
			}

		case <-ctx.Done():
			return nil
		}
	}
}

func pointerToInt32(i int32) *int32 {
	return &i
}

In production you’d usually call leaderelection.RunOrDie from k8s.io/client-go/tools/leaderelection instead of hand-rolling this loop — the code above shows what that package does internally.

Redis-based election with SET NX PX

Redis gives you the atomic primitive directly. The two rules: acquire with SET key token NX PX ttl (single command, atomic), and release/renew only through Lua so the compare-and-delete can’t race another leader:

import redis
import time
import uuid

RELEASE_LUA = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
end
return 0
"""

RENEW_LUA = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("pexpire", KEYS[1], ARGV[2])
end
return 0
"""


class RedisLeaderElection:
    def __init__(self, redis_client, lock_key="leader_lock", ttl_ms=10000):
        self.redis = redis_client
        self.lock_key = lock_key
        self.ttl_ms = ttl_ms
        self.node_id = uuid.uuid4().hex
        self.token = None

    def acquire(self):
        # One atomic command — no check-then-set race.
        self.token = f"{self.node_id}:{int(time.time() * 1000)}"
        return self.redis.set(
            self.lock_key, self.token, nx=True, px=self.ttl_ms
        ) is not None

    def renew(self):
        # Lua: extend only if we still hold it.
        return self.redis.eval(
            RENEW_LUA, 1, self.lock_key, self.token, self.ttl_ms
        ) == 1

    def release(self):
        # Lua: delete only our own token — never someone else's lease.
        self.redis.eval(RELEASE_LUA, 1, self.lock_key, self.token)

Single-instance Redis covers soft coordination fine. For guarantees under failover, Redlock spreads the lease across N independent masters — but it still can’t fence writes, so pair it with a token check wherever the leader writes state.

Runnable versions of these implementations (plus a simulated crash-and-failover demo that needs no services) live in the companion repository.

Frequently Asked Questions

What is the difference between leader election and distributed locking?

Leader election is a specialized lock held for a long period that picks one coordinator; distributed locking protects arbitrary short-lived critical sections. Election adds term semantics, heartbeats, and usually fencing tokens — see the distributed lock pattern for the general form.

Can different subsystems each have their own leader?

Yes, and they usually should. Elect one leader per shard, partition, or job type: coordination stays simple while the work scales out. The risk is a correlated failure electing leaders that all land on one node.

Is leader election enough for consensus?

No. Election picks a coordinator, but it doesn't replicate decisions to followers or guarantee they agree. Consensus algorithms such as Raft or Paxos include leader election and replicated state; election alone is only the first third of Raft.

What is a fencing token and why do I need one?

A monotonic number issued by the store each time a new leader wins. Downstream services remember the highest token seen and reject writes carrying older ones, which is the only reliable way to stop a partitioned or paused ex-leader from still acting as leader.

How long should the lease duration be?

Long enough to survive your worst GC pause or network blip, short enough to meet your failover budget. 15-30 seconds with renewal at TTL/3 covers most deployments; go shorter only if your store supports sub-second heartbeats cheaply.

What happens to the old leader after it loses the lease?

It should discover the loss on its next renewal attempt (the CAS fails or the fencing check rejects it), stop acting as leader, and rejoin as a follower. If it keeps writing anyway, that's exactly the failure fencing tokens exist to catch.