Chaos Engineering

Overview

Chaos engineering means breaking your own system on purpose — kill a pod, add 500ms of latency, drop packets — and watching whether it degrades gracefully or takes everyone down. Netflix did this first with Chaos Monkey, which randomly terminates production instances every working day. The point isn’t the breakage; it’s what the dashboards tell you afterward. Litmus, Gremlin, and AWS Fault Injection Simulator now package the same idea so you don’t have to write your own kill scripts.

This page is the practical companion to the Chaos Engineering Guide: real fault-injection configs, a steady-state checker, and a game day runbook you can copy.

When to Use

Reach for chaos experiments when:

• Your system is distributed enough that a single dependency dying shouldn’t take it down — but you’re not sure it won’t. (Event-Driven Microservices covers the resilient side of that architecture.)
• A disaster-recovery drill or game day is on the calendar and you need concrete experiments, not just a meeting. (Load Testing covers the capacity half of the drill.)
• You’ve built auto-scaling, failover, or self-healing and never actually watched it work. Health checks alone don’t prove recovery — Health Check Endpoint gets the probes right first.
• A launch or high-traffic event is coming and “it should be fine” isn’t a plan. Retry Logic is the other half of graceful degradation.

How an Experiment Works

Every experiment follows the same loop regardless of tool: measure normal behavior, state a falsifiable hypothesis (“if the primary database dies, failover completes in under 30 seconds”), inject a fault with the smallest blast radius that still answers the question, and either confirm the hypothesis or abort, fix, and repeat.

Solution

Kubernetes Pod Chaos (Litmus)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-experiment
spec:
  appinfo:
    appns: 'production'
    applabel: 'app=payment-service'
    appkind: 'deployment'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'

Network Latency Injection (tc + Bash)

#!/bin/bash
# Add 500ms latency to egress traffic on eth0

echo "Injecting 500ms latency for 60 seconds..."
tc qdisc add dev eth0 root netem delay 500ms 50ms distribution normal

sleep 60

echo "Removing latency..."
tc qdisc del dev eth0 root

# Verify with ping
ping -c 5 api.example.com

AWS Fault Injection Simulator (Python)

import boto3

fis = boto3.client('fis')

response = fis.start_experiment(
    experimentTemplateId='EXT-12345678',
    tags={'Environment': 'staging'}
)

print(f"Experiment started: {response['experiment']['id']}")

Gremlin Attack (CPU Stress)

# Install Gremlin CLI
sudo apt-get install gremlin

# CPU stress on a specific container for 60 seconds
gremlin attack cpu --percent 80 --length 60 --container <container-id>

# Network blackhole (drop all traffic to a dependency)
gremlin attack blackhole --ip 10.0.4.20 --length 30

# Memory consumption
gremlin attack memory --percent 75 --length 45

Toxiproxy (Network Chaos for Dependencies)

# Start Toxiproxy
docker run -p 8474:8474 -p 3306:3306 ghcr.io/shopify/toxiproxy:latest

# Create a proxy for MySQL
toxiproxy-cli create mysql-proxy -l 0.0.0.0:3306 -u 127.0.0.1:3307

# Add 2000ms latency to MySQL connections
toxiproxy-cli toxic add mysql-proxy -n latency -t latency=2000

# Add timeout (connections drop after 500ms)
toxiproxy-cli toxic add mysql-proxy -n timeout -t timeout=500

# Remove all toxics
toxiproxy-cli toxic delete mysql-proxy -n latency

Steady-State Verification Script

import requests
import time
import sys

def check_steady_state():
    """Verify SLOs before, during, and after chaos experiments."""
    checks = [
        ("https://api.example.com/health", 200, 500),  # URL, expected_status, max_latency_ms
        ("https://api.example.com/metrics", 200, 1000),
    ]
    all_pass = True
    for url, expected_status, max_latency in checks:
        try:
            start = time.time()
            resp = requests.get(url, timeout=5)
            latency_ms = (time.time() - start) * 1000
            if resp.status_code != expected_status:
                print(f"FAIL: {url} returned {resp.status_code}, expected {expected_status}")
                all_pass = False
            elif latency_ms > max_latency:
                print(f"FAIL: {url} latency {latency_ms:.0f}ms exceeds {max_latency}ms")
                all_pass = False
            else:
                print(f"OK: {url} - {resp.status_code} in {latency_ms:.0f}ms")
        except requests.RequestException as e:
            print(f"FAIL: {url} unreachable: {e}")
            all_pass = False
    return all_pass

if __name__ == "__main__":
    if not check_steady_state():
        print("Steady state violated. Aborting experiment.")
        sys.exit(1)
    print("Steady state confirmed. Safe to proceed.")

• Companion code: the chaos-engineering companion repo packages the fault-injection scripts, the Litmus manifest, the steady-state checker and the game day template.

Explanation

Five chaos experiment types:

1. Infrastructure: Kill VMs, terminate containers, detach volumes
2. Network: Inject latency, drop packets, partition zones
3. Application: Throw exceptions, return 503s, trigger memory leaks
4. State: Fill disks, corrupt databases, expire certificates
5. Dependency: Make downstream APIs timeout or return errors

The blast radius principle:

• Start in staging, then move to production with minimal traffic
• Always have an abort button (automatic rollback on SLO violation)
• Run during business hours when the team is available
• Measure against SLOs, not just “does it crash”

Steady state is the piece most teams skip: if you can’t quantify normal behavior — typical error rate, p99 latency, requests per second — the experiment tells you nothing, because there is no baseline to break. The verification script above exists for exactly that: run it before the fault, during it, and after cleanup, and only accept a “pass” when all three runs look identical.

Variants

Tool	Platform	Experiment Types
Chaos Monkey	AWS/Netflix	Instance termination
Litmus	Kubernetes	Pod, network, disk, stress
Gremlin	Multi-cloud	CPU, memory, network, state
AWS FIS	AWS	EC2, ECS, EKS, RDS failures
Toxiproxy	Any	Network latency, timeouts

Game Day Runbook Template

# Game Day: <Date>

## Participants
- Facilitator: <name>
- Observers: <names>
- On-call engineer: <name>

## Scope
- Target service: payment-service
- Environment: staging
- Blast radius: 10% of traffic

## Experiments
1. Kill primary database pod - expect failover < 30s
2. Inject 500ms latency to payment gateway - expect p99 < 2s
3. Drop 20% of packets to Redis - expect cache miss rate increase only

## Abort Criteria
- Error rate > 1% for > 2 minutes
- p99 latency > 5s
- Any data corruption detected

## Rollback Plan
- `kubectl rollout undo deployment/payment-service`
- `toxiproxy-cli toxic delete --all`
- Page on-call engineer

What Works

• Measure before you break anything. Without a recorded baseline for error rate, latency, and throughput, an experiment produces numbers with nothing to compare against.
• Write the hypothesis down. “If we kill the primary database, failover completes in <30s” is testable; “let’s see what happens” is not an experiment.
• Wire the abort switch to monitoring. The experiment should stop itself the moment error rate crosses your threshold — don’t rely on someone watching a dashboard.
• Schedule game days. A quarterly chaos event with the whole team in the room (facilitator, observers, on-call) builds far more confidence than ad-hoc testing.
• Every experiment changes something. If a run produces no runbook update or architecture fix, you either measured nothing or learned nothing.

Common Mistakes

1. Injecting faults you can’t see. If your dashboards don’t show the blast radius, you’re not experimenting — you’re just breaking things. Get observability right before the first attack.
2. Going straight to production. Staging isn’t a rehearsal, it’s the proof that your experiment is survivable. Earn production chaos with clean staging runs.
3. No way to stop it. An experiment without a working abort is an outage you scheduled yourself. Test the rollback path before you test the failure.
4. Only watching the failure. Half the value is in recovery: does auto-healing actually heal, or does it just file a nice-looking log line?
5. One blast radius: everything. If a single experiment can touch every customer, your radius is too big. Shard it — 10% of traffic, one AZ, one dependency.
6. Running experiments without baselines. You need steady-state metrics before chaos to compare:

# Record baseline before experiment
curl -s https://api.example.com/metrics > baseline-metrics.json
# Run experiment
# Compare after
curl -s https://api.example.com/metrics > post-experiment-metrics.json
diff <(jq '.latency_p99' baseline-metrics.json) <(jq '.latency_p99' post-experiment-metrics.json)

1. Not cleaning up after experiments. Toxiproxy toxics, tc rules, and injected failures persist if not removed:

# Always clean up
tc qdisc del dev eth0 root 2>/dev/null
toxiproxy-cli toxic delete --all
kubectl delete chaosengine --all -n staging

Performance Tips

1. Run experiments in off-peak hours. Even with narrow blast radius, experiments add load:

# Schedule for 2am when traffic is lowest
0 2 * * * /usr/local/bin/chaos-experiment.sh

1. Use short experiment durations. 30-60 seconds is enough to observe behavior:

# Litmus: 30 seconds max
env:
  - name: TOTAL_CHAOS_DURATION
    value: '30'

1. Parallelize steady-state checks. Check multiple endpoints simultaneously:

import concurrent.futures

def check_all_endpoints(urls):
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        results = list(executor.map(check_endpoint, urls))
    return all(results)

See Also

• Principles of Chaos Engineering
• Litmus Chaos docs
• AWS Fault Injection Simulator
• Netflix Chaos Monkey — Tech Blog