Health Endpoint Monitoring Pattern

Overview

The Health Endpoint Monitoring Pattern exposes lightweight endpoints that report whether a service is alive and ready to handle traffic. Load balancers, container orchestrators, and monitoring tools can call these endpoints to decide whether to route traffic to an instance or restart it.

This pattern is the foundation of self-healing systems and is essential for any service that runs in a dynamic environment where instances can fail or restart at any time.

When to Use

Use this pattern when:

• You run services in containers or behind a load balancer
• You want an orchestrator to restart unhealthy instances automatically
• You need to distinguish between “the process is running” and “the service is usable”
• You want to add dependency health checks without modifying client code
• You need to surface health data to a monitoring dashboard or alert system

Solution

// Express health endpoints with liveness and readiness probes
const express = require('express');
const app = express();

app.get('/health/live', (req, res) => {
  res.status(200).json({ status: 'alive' });
});

app.get('/health/ready', async (req, res) => {
  const dbHealthy = await checkDatabaseConnection();
  const cacheHealthy = await checkCacheConnection();
  if (dbHealthy && cacheHealthy) {
    res.status(200).json({ status: 'ready' });
  } else {
    res.status(503).json({ status: 'not ready' });
  }
});

app.listen(3000);

# Kubernetes liveness and readiness probes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    spec:
      containers:
      - name: api
        image: api:latest
        livenessProbe:
          httpGet:
            path: /health/live
            port: 3000
          initialDelaySeconds: 10
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 10

Explanation

Health endpoints separate two concerns:

• Liveness: the process is running and should not be restarted. If liveness fails, the orchestrator kills the container and starts a new one.
• Readiness: the service is ready to accept traffic. If readiness fails, the load balancer stops sending requests but does not restart the instance.

By checking dependencies such as databases, caches, and message queues, readiness probes prevent traffic from reaching an instance that cannot serve requests correctly. This improves reliability and reduces error rates during deployments or outages.

Variants

Endpoint	Purpose	Response
Liveness	Is the process alive?	200 when running, 500 otherwise
Readiness	Can it handle traffic?	200 when dependencies are healthy, 503 otherwise
Startup	Has it finished starting?	200 after initialization is complete
Deep health	Detailed subsystem status	JSON with per-dependency health

What Works

• Keep the liveness probe lightweight and dependency-free
• Make the readiness probe reflect the actual ability to serve requests
• Return consistent status codes (200 for healthy, 503 for unhealthy)
• Avoid heavy operations in health checks to prevent false failures
• Add timeouts and retry budgets for dependency checks
• Log health check failures for debugging but do not spam logs on every call

Common Mistakes

• Using a single endpoint that returns OK even when the service is broken
• Making health checks depend on external services that are not critical
• Returning 500 for liveness, causing unnecessary restarts
• Forgetting to test readiness probes during deployment rollouts
• Exposing health endpoints publicly without authentication or rate limiting

Advanced Solutions

Deep health endpoint with dependency checks

Implement a detailed health endpoint that checks all dependencies:

const express = require('express');
const app = express();

const healthChecks = {
  database: async () => {
    try {
      await pool.query('SELECT 1');
      return { status: 'healthy', latency: Date.now() - start };
    } catch (error) {
      return { status: 'unhealthy', error: error.message };
    }
  },
  cache: async () => {
    try {
      await cache.ping();
      return { status: 'healthy' };
    } catch (error) {
      return { status: 'unhealthy', error: error.message };
    }
  },
  messageQueue: async () => {
    try {
      await channel.checkQueue();
      return { status: 'healthy' };
    } catch (error) {
      return { status: 'unhealthy', error: error.message };
    }
  }
};

app.get('/health/deep', async (req, res) => {
  const results = {};
  let overallHealthy = true;

  for (const [name, check] of Object.entries(healthChecks)) {
    try {
      const start = Date.now();
      const result = await check();
      results[name] = { ...result, checkTime: Date.now() - start };
      if (result.status !== 'healthy') {
        overallHealthy = false;
      }
    } catch (error) {
      results[name] = { status: 'error', error: error.message };
      overallHealthy = false;
    }
  }

  res.status(overallHealthy ? 200 : 503).json({
    status: overallHealthy ? 'healthy' : 'unhealthy',
    checks: results,
    timestamp: new Date().toISOString()
  });
});

Startup probe for slow-initializing services

Use a startup probe for services that take time to initialize:

# Kubernetes deployment with startup probe
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slow-startup-service
spec:
  template:
    spec:
      containers:
      - name: app
        image: app:latest
        startupProbe:
          httpGet:
            path: /health/startup
            port: 3000
          initialDelaySeconds: 0
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 30  # Allow up to 5 minutes to start
        livenessProbe:
          httpGet:
            path: /health/live
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 3000
          initialDelaySeconds: 10
          periodSeconds: 10

// Startup endpoint that returns success only after initialization
let isInitialized = false;

async function initialize() {
  // Perform slow initialization tasks
  await loadConfiguration();
  await warmUpCache();
  await connectToExternalServices();
  isInitialized = true;
}

app.get('/health/startup', (req, res) => {
  if (isInitialized) {
    res.status(200).json({ status: 'initialized' });
  } else {
    res.status(503).json({ status: 'initializing' });
  }
});

// Start initialization in background
initialize();

Health endpoint with circuit breaker

Add circuit breaker pattern to prevent health check storms:

class HealthCheckCircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.failureCount = 0;
    this.lastFailureTime = null;
    this.threshold = threshold;
    this.timeout = timeout;
    this.state = 'closed'; // closed, open, half-open
  }

  recordSuccess() {
    this.failureCount = 0;
    this.state = 'closed';
  }

  recordFailure() {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    
    if (this.failureCount >= this.threshold) {
      this.state = 'open';
    }
  }

  shouldAllowCheck() {
    if (this.state === 'closed') return true;
    
    if (this.state === 'open') {
      const timeSinceLastFailure = Date.now() - this.lastFailureTime;
      if (timeSinceLastFailure > this.timeout) {
        this.state = 'half-open';
        return true;
      }
      return false;
    }
    
    return true;
  }
}

const circuitBreaker = new HealthCheckCircuitBreaker();

app.get('/health/ready', async (req, res) => {
  if (!circuitBreaker.shouldAllowCheck()) {
    return res.status(503).json({ status: 'circuit open' });
  }

  try {
    const dbHealthy = await checkDatabaseConnection();
    const cacheHealthy = await checkCacheConnection();
    
    if (dbHealthy && cacheHealthy) {
      circuitBreaker.recordSuccess();
      res.status(200).json({ status: 'ready' });
    } else {
      circuitBreaker.recordFailure();
      res.status(503).json({ status: 'not ready' });
    }
  } catch (error) {
    circuitBreaker.recordFailure();
    res.status(503).json({ status: 'error', message: error.message });
  }
});