Health Check Pattern: Expose Liveness and Readiness Probes

Overview

Health checks expose endpoints that tell the orchestrator whether an application is running and ready to serve traffic. Kubernetes, ECS, and other orchestrators use two distinct probe types: liveness (is the process alive?) and readiness (can the process handle requests?). Liveness probes determine if a container should be restarted. Readiness probes determine if a container should receive traffic. Confusing the two leads to unnecessary restarts or traffic sent to unready instances.

This pattern covers the orchestrator’s side of the contract — probe lifecycle, timing, thresholds, and shutdown behavior. If you need the endpoints as a general monitoring surface for dashboards and external checks, that’s the Health Endpoint Monitoring Pattern.

When to Use

• Any application running in a container orchestrator (Kubernetes, ECS, Nomad)
• Services with dependencies that take time to initialize (database connections, cache warmup)
• Applications that can become unresponsive without crashing (deadlocks, thread exhaustion)
• Services that need to drain connections before shutting down
• Load balancer health checks for traffic routing

When NOT to Use

• Simple scripts or batch jobs — no long-running process to probe
• Applications without external dependencies — a simple process check suffices
• When the orchestrator doesn’t support probes — use a process manager instead
• For internal health monitoring — use metrics and alerting, not probes

Solution

Basic health endpoint (Python/FastAPI)

# Python — basic health check endpoint
from fastapi import FastAPI, status
from fastapi.responses import JSONResponse

app = FastAPI()

@app.get("/health")
async def health():
    """Basic liveness check — process is alive."""
    return {"status": "healthy"}

@app.get("/health/live")
async def liveness():
    """Liveness probe — is the process running?"""
    return {"status": "alive"}

@app.get("/health/ready")
async def readiness():
    """Readiness probe — can we serve traffic?"""
    checks = await run_dependency_checks()
    all_healthy = all(checks.values())

    if not all_healthy:
        return JSONResponse(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            content={"status": "not ready", "checks": checks},
        )

    return {"status": "ready", "checks": checks}

async def run_dependency_checks():
    return {
        "database": await check_database(),
        "redis": await check_redis(),
        "message_queue": await check_message_queue(),
    }

Dependency checks with timeouts

# Python — dependency checks with timeouts
import asyncio
import aio_pika
import asyncpg
import redis.asyncio as aioredis

async def check_database(timeout: float = 2.0) -> bool:
    try:
        async with asyncpg.create_pool("postgresql://localhost/myapp") as pool:
            async with pool.acquire() as conn:
                await asyncio.wait_for(
                    conn.execute("SELECT 1"),
                    timeout=timeout,
                )
        return True
    except Exception:
        return False

async def check_redis(timeout: float = 1.0) -> bool:
    try:
        client = aioredis.from_url("redis://localhost:6379")
        pong = await asyncio.wait_for(client.ping(), timeout=timeout)
        await client.aclose()
        return pong
    except Exception:
        return False

async def check_message_queue(timeout: float = 2.0) -> bool:
    try:
        # Check if we can connect to RabbitMQ
        connection = await asyncio.wait_for(
            aio_pika.connect_robust("amqp://localhost"),
            timeout=timeout,
        )
        await connection.close()
        return True
    except Exception:
        return False

Kubernetes probe configuration

# Kubernetes deployment with liveness and readiness probes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: order-service
          image: order-service:1.0.0
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 2
          startupProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            failureThreshold: 30
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"

Express.js health checks

// JavaScript — Express health check endpoints
const express = require('express');
const app = express();

// Liveness — process is alive
app.get('/health/live', (req, res) => {
  res.json({ status: 'alive', uptime: process.uptime() });
});

// Readiness — can serve traffic
app.get('/health/ready', async (req, res) => {
  // Run dependency checks in parallel — sequential awaits would
  // stack each check's latency and risk tripping the probe timeout
  const [database, redisOk, queue] = await Promise.all([
    checkDatabase(),
    checkRedis(),
    checkQueue(),
  ]);
  const checks = { database, redis: redisOk, queue };

  const allHealthy = Object.values(checks).every(v => v === true);

  if (allHealthy) {
    res.json({ status: 'ready', checks });
  } else {
    res.status(503).json({ status: 'not ready', checks });
  }
});

// Comprehensive health endpoint
app.get('/health', async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    queue: await checkQueue(),
    diskSpace: checkDiskSpace(),
    memoryUsage: checkMemoryUsage(),
  };

  const status = Object.values(checks).every(v => v === true)
    ? 'healthy'
    : 'degraded';

  res.json({
    status,
    uptime: process.uptime(),
    timestamp: new Date().toISOString(),
    checks,
  });
});

async function checkDatabase() {
  try {
    await db.raw('SELECT 1');
    return true;
  } catch {
    return false;
  }
}

async function checkRedis() {
  try {
    await redis.ping();
    return true;
  } catch {
    return false;
  }
}

function checkMemoryUsage() {
  const used = process.memoryUsage().rss;
  const max = 512 * 1024 * 1024; // 512MB limit
  return used < max;
}

Java Spring Boot health checks

Spring Boot Actuator ships liveness and readiness state groups out of the box; the recipe Custom Health Checks with Spring Boot Actuator walks through wiring them. The custom indicators look like this:

// Java — Spring Boot Actuator health indicators
import org.springframework.boot.actuate.health.*;
import org.springframework.stereotype.Component;

@Component
public class DatabaseHealthIndicator extends AbstractHealthIndicator {

    private final DataSource dataSource;

    public DatabaseHealthIndicator(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @Override
    protected void doHealthCheck(Health.Builder builder) {
        try (Connection conn = dataSource.getConnection()) {
            try (Statement stmt = conn.createStatement()) {
                stmt.execute("SELECT 1");
            }
            builder.up().withDetail("database", "PostgreSQL").withDetail("status", "connected");
        } catch (SQLException e) {
            builder.down().withDetail("error", e.getMessage());
        }
    }
}

@Component
public class RedisHealthIndicator extends AbstractHealthIndicator {

    private final RedisTemplate<String, String> redisTemplate;

    public RedisHealthIndicator(RedisTemplate<String, String> redisTemplate) {
        this.redisTemplate = redisTemplate;
    }

    @Override
    protected void doHealthCheck(Health.Builder builder) {
        try {
            String response = redisTemplate.getConnectionFactory()
                .getConnection().ping();
            builder.up().withDetail("redis", response);
        } catch (Exception e) {
            builder.down().withDetail("error", e.getMessage());
        }
    }
}

// application.yml — configure health endpoints
// management:
//   endpoints:
//     web:
//       exposure:
//         include: health,info,metrics
//   endpoint:
//     health:
//       probes:
//         enabled: true
//       show-details: always
//   health:
//     livenessstate:
//       enabled: true
//     readinessstate:
//       enabled: true

Graceful shutdown with health checks

# Python — graceful shutdown with readiness probe
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.responses import JSONResponse

shutting_down = False

@asynccontextmanager
async def lifespan(app: FastAPI):
    yield
    # Runs on shutdown (SIGTERM from Kubernetes lands here)
    global shutting_down
    shutting_down = True
    # Give the kube-proxy/endpoints controller a moment to observe
    # readiness=503 before the server stops accepting connections
    await asyncio.sleep(5)
    await drain_connections(timeout=25)

app = FastAPI(lifespan=lifespan)

@app.get("/health/ready")
async def readiness():
    if shutting_down:
        return JSONResponse(
            status_code=503,
            content={"status": "shutting down"},
        )
    # Normal readiness check
    checks = await run_dependency_checks()
    if not all(checks.values()):
        return JSONResponse(status_code=503, content={"status": "not ready", "checks": checks})
    return {"status": "ready", "checks": checks}

async def drain_connections(timeout: int = 25):
    """Wait for in-flight requests to complete."""
    deadline = asyncio.get_event_loop().time() + timeout
    while getattr(app.state, "active_requests", 0) > 0:
        if asyncio.get_event_loop().time() > deadline:
            break
        await asyncio.sleep(0.5)

Startup probe for slow initializers

# Python — startup probe for applications with slow initialization
import time
from fastapi import FastAPI

app = FastAPI()
startup_time = time.time()
ready_since = None

@app.get("/health/startup")
async def startup_probe():
    """Startup probe — is the app still initializing?"""
    if ready_since is None:
        return JSONResponse(status_code=503, content={"status": "starting"})
    return {"status": "started", "ready_since": ready_since}

@app.get("/health/ready")
async def readiness():
    global ready_since
    if ready_since is None:
        # Check if initialization is complete
        if await is_initialization_complete():
            ready_since = time.time()
        else:
            return JSONResponse(status_code=503, content={"status": "initializing"})
    # Normal readiness check
    return {"status": "ready"}

Composite health check with circuit breakers

// JavaScript — composite health with circuit breaker states
const CircuitBreaker = require('opossum');

const dbBreaker = new CircuitBreaker(queryDatabase, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
});

app.get('/health/ready', async (req, res) => {
  const checks = {};

  // Database check via circuit breaker
  try {
    await dbBreaker.fire('SELECT 1');
    checks.database = {
      status: dbBreaker.opened ? 'degraded' : 'healthy',
      circuitState: dbBreaker.status,
    };
  } catch {
    checks.database = {
      status: 'unhealthy',
      circuitState: dbBreaker.status,
    };
  }

  const allHealthy = Object.values(checks)
    .every(c => c.status === 'healthy');

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'ready' : 'degraded',
    checks,
  });
});

Explanation

The kubelet — not a central controller — runs every probe against each container. Each probe type answers a different question and triggers a different action on failure:

Probe	Question it answers	What Kubernetes does on failure
startupProbe	Has the container finished booting?	Keeps probing until failureThreshold × periodSeconds elapses, then restarts the container. While it runs, liveness and readiness are paused.
readinessProbe	Can the container serve traffic right now?	Removes the pod from the Service endpoints. The process keeps running — no restart.
livenessProbe	Is the process alive or deadlocked?	Restarts the container after failureThreshold consecutive failures.

Two consequences follow from this model. First, a failing readiness probe is cheap — the pod just stops receiving requests until it recovers, which makes it the right place for dependency checks. Second, a failing liveness probe is expensive — the container is killed and restarted, losing in-flight work and warm caches. That asymmetry is why liveness should only test whether the process itself is stuck, never whether a database or downstream API is reachable.

startupProbe exists because initialDelaySeconds forces a bad trade-off: set it too low and slow boots get killed mid-initialization; set it too high and genuinely dead containers take minutes to restart. The startup probe replaces the guess with a loop — failureThreshold: 30 and periodSeconds: 5 gives the app 150 seconds to boot, after which the normal probes take over at full speed.

Health checks also have a cost budget. A readiness probe that calls three dependencies every 5 seconds generates real load on your database and Redis: for 10 replicas that’s 6 connections per second per dependency. Keep dependency checks cheap (SELECT 1, PING), bound them with short timeouts, and don’t let a slow dependency consume your whole probe budget — that’s what the circuit breaker variant above is for.

Variants

Health check with dependency graph

# Python — health check with dependency graph
class HealthChecker:
    def __init__(self):
        self.dependencies = {}

    def register(self, name, check_fn, dependencies=None):
        self.dependencies[name] = {
            "check": check_fn,
            "deps": dependencies or [],
        }

    async def check_all(self):
        results = {}
        for name, config in self.dependencies.items():
            # Check dependencies first
            dep_results = {dep: results.get(dep, False) for dep in config["deps"]}
            if not all(dep_results.values()):
                results[name] = {
                    "status": "skipped",
                    "reason": f"Dependencies unhealthy: {[d for d, v in dep_results.items() if not v]}",
                }
                continue

            try:
                healthy = await config["check"]()
                results[name] = {"status": "healthy" if healthy else "unhealthy"}
            except Exception as e:
                results[name] = {"status": "unhealthy", "error": str(e)}

        return results

# Usage
checker = HealthChecker()
checker.register("database", check_database)
checker.register("redis", check_redis)
checker.register("cache", check_cache, dependencies=["redis"])
checker.register("api", check_api, dependencies=["database", "cache"])

TCP-based health check

# Kubernetes — TCP socket probe for non-HTTP services
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: tcp-service
      image: tcp-service:1.0
      livenessProbe:
        tcpSocket:
          port: 5432
        initialDelaySeconds: 15
        periodSeconds: 10
      readinessProbe:
        tcpSocket:
          port: 5432
        initialDelaySeconds: 5
        periodSeconds: 5

Command-based health check

For worker processes without an HTTP server, use an exec probe that runs a script inside the container. The same idea powers Docker’s HEALTHCHECK instruction — see Docker Health Check Configuration for the Docker-native variant.

# Python — command-based probe for worker processes
# probe_script.py
import sys
import psycopg2

def check_worker_health():
    try:
        conn = psycopg2.connect("postgresql://localhost/myapp")
        cur = conn.cursor()
        # Cheap connectivity probe — if the query runs, the worker is fine
        cur.execute("SELECT COUNT(*) FROM jobs WHERE status = 'processing'")
        cur.fetchone()
        cur.close()
        conn.close()
        return True
    except Exception:
        return False

if __name__ == "__main__":
    if check_worker_health():
        print("healthy")
        sys.exit(0)
    else:
        print("unhealthy")
        sys.exit(1)

# Kubernetes — command-based probe
spec:
  containers:
    - name: worker
      image: worker:1.0
      livenessProbe:
        exec:
          command:
            - python
            - /app/probe_script.py
        initialDelaySeconds: 30
        periodSeconds: 60

Best Practices

• For a deeper guide, see Health Endpoint Monitoring Pattern.

• Separate liveness and readiness — liveness checks if the process is alive, readiness checks if it can serve traffic

• Keep liveness simple — don’t check dependencies in liveness, only the process itself

• Use startup probes for slow initializers — prevents premature restarts during boot

• Set reasonable timeouts — 2-3 seconds for HTTP probes, longer for startup

• Return 503 when not ready — orchestrators use HTTP status codes, not response body

• Include dependency details in readiness — helps debug why a service isn’t ready

• Implement graceful shutdown — return 503 from readiness during shutdown to stop traffic

• Don’t over-check — checking every second is excessive; every 5-10 seconds is sufficient

• Use circuit breakers in health checks — don’t let a slow dependency make your health check timeout

Common Mistakes

• Same endpoint for liveness and readiness: if readiness fails (database down), liveness also fails, causing unnecessary restarts. Use separate endpoints.
• Checking dependencies in liveness: liveness should only verify the process is running. Dependency checks belong in readiness.
• No startup probe: slow-starting applications get killed before they finish initializing. Use startup probes with a high failureThreshold.
• Probe too aggressive: checking every 1 second with a 1-second timeout causes false failures under load. Use 5-10 second intervals.
• Not handling graceful shutdown: during shutdown, the process is alive but can’t serve traffic. Readiness should return 503 immediately.

See Also

• Companion repository — runnable FastAPI demo with all three probe endpoints, a Kubernetes deployment, and a docker-compose lab with PostgreSQL and Redis
• Configure Liveness, Readiness and Startup Probes — official Kubernetes documentation
• Spring Boot Actuator endpoints — built-in liveness and readiness state groups
• Container probes — probe semantics and field reference