Graceful Shutdown: Drain In-Flight Requests Before Exit

Overview

Kill a server mid-request and the client eats a 502 while the database holds half-committed work. Graceful shutdown fixes the exit sequence: on SIGTERM the process stops accepting new work, finishes the requests already in flight, closes database connections, flushes buffers, and only then exits.

Kubernetes makes the timing concrete. Deleting a pod runs the preStop hook first, then fires SIGTERM — and the grace-period clock (30 seconds by default, via terminationGracePeriodSeconds) is already ticking. Whatever is still alive past the budget gets SIGKILL mid-response.

When to Use

• Any long-running server (HTTP, gRPC, WebSocket)
• Kubernetes deployments where pods are regularly terminated
• Applications with background workers processing jobs
• Services with open database connections or file handles
• Message consumers that need to finish processing current messages

When NOT to Use

• CLI tools and scripts that exit after completion
• Serverless functions where the platform handles lifecycle
• Batch jobs that are designed to be killed and restarted

Solution

Python graceful shutdown with signal handling

# shutdown/graceful.py — Graceful shutdown for Python HTTP servers
import signal
import threading
import time
import sys
from http.server import HTTPServer, BaseHTTPRequestHandler

class GracefulHTTPServer(HTTPServer):
    """HTTP server with graceful shutdown support."""

    def __init__(self, addr, handler):
        super().__init__(addr, handler)
        self._shutting_down = False
        self._active_requests = 0
        self._lock = threading.Lock()
        self._shutdown_event = threading.Event()

    @property
    def is_shutting_down(self):
        return self._shutting_down

    def request_started(self):
        with self._lock:
            self._active_requests += 1

    def request_finished(self):
        with self._lock:
            self._active_requests -= 1
            if self._shutting_down and self._active_requests == 0:
                self._shutdown_event.set()

    def graceful_shutdown(self, timeout=30):
        """Stop accepting new requests, wait for active ones to finish."""
        print("Starting graceful shutdown...")
        self._shutting_down = True

        # Stop accepting new connections
        self.socket.close()

        # Wait for active requests to complete
        with self._lock:
            if self._active_requests == 0:
                print("No active requests, shutting down immediately")
                return

            print(f"Waiting for {self._active_requests} active requests...")

        if self._shutdown_event.wait(timeout=timeout):
            print("All requests completed, shutting down")
        else:
            print(f"Timeout after {timeout}s, forcing shutdown with "
                  f"{self._active_requests} requests still active")

        # Cleanup resources
        self._cleanup()

    def _cleanup(self):
        """Close database connections, flush buffers, etc."""
        print("Cleaning up resources...")
        # Close database connections
        # Flush log buffers
        # Close file handles
        print("Cleanup complete")


class GracefulRequestHandler(BaseHTTPRequestHandler):
    """Request handler that tracks active requests for graceful shutdown."""

    def handle_one_request(self):
        self.server.request_started()
        try:
            super().handle_one_request()
        finally:
            self.server.request_finished()

    def do_GET(self):
        if self.server.is_shutting_down:
            self.send_response(503)
            self.send_header("Connection", "close")
            self.end_headers()
            self.wfile.write(b'{"error": "Server shutting down"}')
            return

        # Simulate work
        time.sleep(0.5)
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'{"status": "ok"}')


# Setup signal handlers
server = GracefulHTTPServer(("0.0.0.0", 8080), GracefulRequestHandler)

def signal_handler(signum, frame):
    server.graceful_shutdown(timeout=30)
    sys.exit(0)

signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler)

print("Server running on port 8080")
server.serve_forever()

Node.js graceful shutdown

// shutdown/graceful.js — Graceful shutdown for Node.js Express
const express = require("express");
const http = require("http");

const app = express();
const server = http.createServer(app);

let isShuttingDown = false;
let activeRequests = 0;

// Track active requests
app.use((req, res, next) => {
    if (isShuttingDown) {
        res.setHeader("Connection", "close");
        return res.status(503).json({ error: "Server shutting down" });
    }
    activeRequests++;
    res.on("finish", () => activeRequests--);
    next();
});

app.get("/health", (req, res) => {
    if (isShuttingDown) {
        return res.status(503).json({ status: "shutting_down" });
    }
    res.json({ status: "healthy" });
});

app.get("/api/data", (req, res) => {
    setTimeout(() => {
        res.json({ data: "response", activeRequests });
    }, 500);
});

// Graceful shutdown
function gracefulShutdown(signal) {
    console.log(`Received ${signal}, starting graceful shutdown...`);
    isShuttingDown = true;

    // Stop accepting new connections
    server.close(() => {
        console.log("HTTP server closed");
    });

    // Wait for active requests
    const checkInterval = setInterval(() => {
        if (activeRequests === 0) {
            clearInterval(checkInterval);
            console.log("All requests completed, exiting");
            cleanup();
        }
    }, 100);

    // Force exit after timeout
    setTimeout(() => {
        console.log(`Timeout: ${activeRequests} requests still active, forcing exit`);
        cleanup();
    }, 30000);
}

function cleanup() {
    // Close database connections
    // Flush log buffers
    // Close message queue connections
    console.log("Cleanup complete, exiting");
    process.exit(0);
}

process.on("SIGTERM", () => gracefulShutdown("SIGTERM"));
process.on("SIGINT", () => gracefulShutdown("SIGINT"));

server.listen(8080, () => console.log("Server running on port 8080"));

Java graceful shutdown with Spring

// ShutdownConfig.java — Spring Boot graceful shutdown
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory;
import org.springframework.context.annotation.Configuration;
import org.springframework.boot.web.server.Shutdown;

@Configuration
public class GracefulShutdownConfig
    implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

    @Override
    public void customize(TomcatServletWebServerFactory factory) {
        factory.setShutdown(Shutdown.GRACEFUL);
        // Spring will wait for active requests to complete
    }
}

// application.properties
// server.shutdown=graceful
// spring.lifecycle.timeout-per-shutdown-phase=30s

// Custom shutdown hook for resource cleanup
import org.springframework.context.SmartLifecycle;
import org.springframework.stereotype.Component;

@Component
public class ResourceCleanupHook implements SmartLifecycle {

    private volatile boolean running = false;

    @Override
    public void start() {
        running = true;
    }

    @Override
    public void stop() {
        running = false;
        // Close database connections
        closeDatabaseConnections();
        // Flush log buffers
        flushLogBuffers();
        // Close message consumer connections
        closeMessageConsumers();
        System.out.println("Resource cleanup complete");
    }

    @Override
    public boolean isRunning() {
        return running;
    }

    @Override
    public int getPhase() {
        // Run cleanup early in shutdown sequence
        return Integer.MIN_VALUE + 100;
    }

    private void closeDatabaseConnections() {
        System.out.println("Closing database connections...");
    }

    private void flushLogBuffers() {
        System.out.println("Flushing log buffers...");
    }

    private void closeMessageConsumers() {
        System.out.println("Closing message consumers...");
    }
}

Kubernetes graceful shutdown

# k8s-graceful-shutdown.yaml — Kubernetes pod with graceful shutdown
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      terminationGracePeriodSeconds: 45  # Give app 45s to drain
      containers:
        - name: api
          image: shop/api-server:latest
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                # Sleep to let load balancer remove this pod
                # before SIGTERM is sent
                command: ["sh", "-c", "sleep 5"]
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: SHUTDOWN_TIMEOUT
              value: "30"
---
# Service with proper health check removal
apiVersion: v1
kind: Service
metadata:
  name: api-server
spec:
  selector:
    app: api-server
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

Python background worker graceful shutdown

# shutdown/worker.py — Graceful shutdown for background workers
import signal
import threading
import time
import queue

class GracefulWorker:
    """Background worker that finishes current job before exiting."""

    def __init__(self, job_queue, max_workers=4):
        self._queue = job_queue
        self._max_workers = max_workers
        self._workers = []
        self._active_jobs = 0
        self._lock = threading.Lock()
        self._shutdown = threading.Event()
        self._jobs_done = threading.Event()

    def start(self):
        for i in range(self._max_workers):
            t = threading.Thread(target=self._worker_loop, args=(i,), daemon=True)
            t.start()
            self._workers.append(t)

    def _worker_loop(self, worker_id):
        while not self._shutdown.is_set():
            try:
                job = self._queue.get(timeout=1)
            except queue.Empty:
                continue

            with self._lock:
                self._active_jobs += 1

            try:
                self._process_job(job, worker_id)
            except Exception as e:
                print(f"Worker {worker_id} job failed: {e}")
            finally:
                with self._lock:
                    self._active_jobs -= 1
                    if self._shutdown.is_set() and self._active_jobs == 0:
                        self._jobs_done.set()
                self._queue.task_done()

    def _process_job(self, job, worker_id):
        print(f"Worker {worker_id} processing job {job['id']}")
        time.sleep(2)  # Simulate work
        print(f"Worker {worker_id} completed job {job['id']}")

    def shutdown(self, timeout=30):
        """Stop accepting new jobs, finish active ones."""
        print("Worker shutdown initiated...")
        self._shutdown.set()

        with self._lock:
            if self._active_jobs == 0:
                print("No active jobs, exiting immediately")
                return

        if self._jobs_done.wait(timeout=timeout):
            print("All jobs completed, exiting")
        else:
            print(f"Timeout: {self._active_jobs} jobs still active")

# Usage
job_queue = queue.Queue()
worker = GracefulWorker(job_queue, max_workers=4)
worker.start()

# Add jobs
for i in range(10):
    job_queue.put({"id": i, "data": f"job-{i}"})

# Handle shutdown
def handle_signal(signum, frame):
    worker.shutdown(timeout=30)

signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)

Load balancer deregistration before shutdown

# shutdown/deregister.py — Deregister from load balancer before draining
import requests
import time
import signal
import sys

class GracefulService:
    """Deregisters from load balancer, then drains requests."""

    def __init__(self, lb_registry_url, health_port=8080):
        self.lb_url = lb_registry_url
        self.health_port = health_port
        self._healthy = True
        self._active_requests = 0

    def deregister(self):
        """Tell the load balancer to stop sending traffic."""
        try:
            resp = requests.post(
                f"{self.lb_url}/deregister",
                json={"port": self.health_port},
                timeout=5
            )
            print(f"Deregistered from load balancer: {resp.status_code}")
        except Exception as e:
            print(f"Deregistration failed: {e}")

        # Wait for LB to propagate
        time.sleep(5)

    def health_check(self):
        """Return 200 if healthy, 503 if shutting down."""
        if not self._healthy:
            return 503
        return 200

    def shutdown(self, timeout=30):
        # Step 1: Mark as unhealthy (LB stops sending new traffic)
        self._healthy = False
        print("Marked as unhealthy")

        # Step 2: Deregister from load balancer
        self.deregister()

        # Step 3: Wait for active requests
        start = time.time()
        while self._active_requests > 0:
            if time.time() - start > timeout:
                print(f"Timeout with {self._active_requests} requests active")
                break
            time.sleep(0.5)

        # Step 4: Cleanup and exit
        print("Shutting down")
        sys.exit(0)

How It Works

Graceful shutdown is an ordered sequence, and the order matters more than the code:

Phase order. Mark yourself unhealthy first so the load balancer and kube-proxy stop routing traffic, then close the listener, then drain, then clean up resources. Reversing the order — closing connections before the LB knows — is how you get a burst of 502s on every deploy.

The drain budget. terminationGracePeriodSeconds should exceed your slowest real request plus the LB propagation delay. If your p99 is 20s and the LB needs ~5s to remove you, a 30s budget cuts it close; 45s gives the drain room to finish.

Trade-offs. A longer grace period means safer shutdowns but slower rollouts and scale-downs. Skipping the preStop sleep saves seconds but lets the LB route requests to a pod that’s already draining. Draining forever isn’t an option either — a hung query or an open SSE stream has to hit a deadline eventually.

For a complete implementation walkthrough with Flask, Go, and zero-downtime restart examples, see the graceful shutdown recipe. If the drain logic lives outside your app process — a companion container that handles deregistration and cleanup — that’s the sidecar pattern.

Companion code: the graceful-shutdown-pattern companion repo has all the examples on this page as runnable files, ready to test with python or node.

Variants

Phased shutdown

# shutdown/phased.py — Multi-phase shutdown with ordered cleanup
class PhasedShutdown:
    """Shuts down in phases: stop accepting, drain, cleanup, exit.
    Each phase has its own timeout."""

    def __init__(self):
        self._phases = [
            ("stop_accepting", self._stop_accepting, 5),
            ("drain_requests", self._drain_requests, 20),
            ("close_connections", self._close_connections, 5),
            ("flush_buffers", self._flush_buffers, 3),
            ("cleanup", self._cleanup, 2),
        ]

    def _stop_accepting(self):
        print("Phase 1: Stop accepting new requests")

    def _drain_requests(self):
        print("Phase 2: Drain active requests")

    def _close_connections(self):
        print("Phase 3: Close database and MQ connections")

    def _flush_buffers(self):
        print("Phase 4: Flush log and metric buffers")

    def _cleanup(self):
        print("Phase 5: Final cleanup")

    def execute(self):
        for name, handler, timeout in self._phases:
            print(f"Starting phase: {name} (timeout: {timeout}s)")
            handler()
        print("All phases complete, exiting")

WebSocket graceful shutdown

// shutdown/websocket.js — Graceful shutdown for WebSocket server
const { WebSocketServer } = require("ws");

const wss = new WebSocketServer({ port: 8080 });
let isShuttingDown = false;

wss.on("connection", (ws) => {
    if (isShuttingDown) {
        ws.close(1001, "Server shutting down");
        return;
    }

    ws.on("message", (data) => {
        ws.send(`Echo: ${data}`);
    });
});

function gracefulShutdown() {
    console.log("WebSocket graceful shutdown...");
    isShuttingDown = true;

    // Close all connections with "going away" code
    wss.clients.forEach((ws) => {
        ws.close(1001, "Server shutting down");
    });

    // Close server after all clients disconnect
    setTimeout(() => {
        wss.close(() => {
            console.log("WebSocket server closed");
            process.exit(0);
        });
    }, 5000);
}

process.on("SIGTERM", gracefulShutdown);

Best Practices

• Fail the health check before you stop the listener — return 503 so the LB pulls you out of rotation while you still have connections to drain
• In Kubernetes, add a preStop hook that sleeps 5–10s — kube-proxy and external LBs take a few seconds to propagate endpoint removal
• Set terminationGracePeriodSeconds ~10–15s above your app’s internal drain timeout so the app decides when to exit, not the kubelet
• Cap the drain — a hung request or open WebSocket must hit a deadline, never block exit forever
• Release resources in dependency order: stop consumers, finish in-flight jobs, close DB pools, flush logs and metrics last
• Re-arm SIGTERM in the handler so a second signal force-exits a stuck drain
• Test under load — run kubectl rollout restart while k6 or hey is hitting the service and confirm zero 5xx

Common Mistakes

• No graceful shutdown at all: the process dies mid-request and every in-flight call turns into a 502.
• Missing preStop delay: kube-proxy and external LBs keep routing traffic for a few seconds after the pod starts terminating.
• Unbounded drain: one hung request or a persistent SSE/WebSocket connection stalls the whole shutdown until SIGKILL.
• Leaked connections: skipping pool cleanup leaves idle sockets behind; repeated deploys exhaust max_connections.
• Health check stays green: the LB keeps sending traffic to an instance that’s already exiting.

See Also

• Kubernetes — Termination of Pods — official documentation on the termination grace period, preStop hooks, and endpoint removal.
• Spring Boot — Graceful Shutdown — official Spring Boot reference for server.shutdown=graceful and spring.lifecycle.timeout-per-shutdown-phase.