Chaos Engineering

Overview

Chaos engineering is the discipline of experimenting on distributed systems to build confidence in their resilience. By intentionally injecting failures — killing instances, injecting latency, corrupting packets — teams discover weaknesses before customers do. Netflix pioneered this with Chaos Monkey; today, tools like Litmus, Gremlin, and AWS Fault Injection Simulator make it accessible to any team.

When to Use

Use this resource when:

• Operating distributed systems where failures are inevitable. See Event-Driven Microservices for resilient architectures.
• Preparing for disaster recovery drills and game days. See Load Testing for capacity verification.
• Validating auto-scaling, failover, and self-healing mechanisms. See Health Check Endpoint for probe configuration.
• Building confidence before high-traffic events (launches, Black Friday). See Retry Logic for handling failures gracefully.

Solution

Kubernetes Pod Chaos (Litmus)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-experiment
spec:
  appinfo:
    appns: 'production'
    applabel: 'app=payment-service'
    appkind: 'deployment'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'

Network Latency Injection (tc + Bash)

#!/bin/bash
# Add 500ms latency to egress traffic on eth0

echo "Injecting 500ms latency for 60 seconds..."
tc qdisc add dev eth0 root netem delay 500ms 50ms distribution normal

sleep 60

echo "Removing latency..."
tc qdisc del dev eth0 root

# Verify with ping
ping -c 5 api.example.com

AWS Fault Injection Simulator (Python)

import boto3

fis = boto3.client('fis')

response = fis.start_experiment(
    experimentTemplateId='EXT-12345678',
    tags={'Environment': 'staging'}
)

print(f"Experiment started: {response['experiment']['id']}")

Explanation

Five chaos experiment types:

1. Infrastructure: Kill VMs, terminate containers, detach volumes
2. Network: Inject latency, drop packets, partition zones
3. Application: Throw exceptions, return 503s, trigger memory leaks
4. State: Fill disks, corrupt databases, expire certificates
5. Dependency: Make downstream APIs timeout or return errors

The blast radius principle:

• Start in staging, then move to production with minimal traffic
• Always have an abort button (automatic rollback on SLO violation)
• Run during business hours when the team is available
• Measure against SLOs, not just “does it crash”

Variants

Tool	Platform	Experiment Types
Chaos Monkey	AWS/Netflix	Instance termination
Litmus	Kubernetes	Pod, network, disk, stress
Gremlin	Multi-cloud	CPU, memory, network, state
AWS FIS	AWS	EC2, ECS, EKS, RDS failures
Toxiproxy	Any	Network latency, timeouts

What Works

• Define steady state first: Know your normal error rate, latency, and throughput
• Hypothesis-driven: “If we kill the primary database, failover completes in <30s”
• Automate rollback: Stop experiments automatically if error rate exceeds 1%
• Run game days: Quarterly scheduled chaos events with the whole team
• Document findings: Every experiment produces a runbook update or architecture fix

Common Mistakes

1. Chaos without monitoring: You can’t observe effects if dashboards are incomplete
2. Production first: Never run chaos in production before proving it safe in staging
3. No rollback plan: Experiments that can’t be stopped quickly become outages
4. Testing only failures: Also test recovery (does auto-healing actually heal?)
5. Ignoring blast radius: One experiment shouldn’t affect all customers

Additional Common Mistakes

1. Running experiments without baselines. You need steady-state metrics before chaos to compare:

# Record baseline before experiment
curl -s https://api.example.com/metrics > baseline-metrics.json
# Run experiment
# Compare after
curl -s https://api.example.com/metrics > post-experiment-metrics.json
diff <(jq '.latency_p99' baseline-metrics.json) <(jq '.latency_p99' post-experiment-metrics.json)

1. Not cleaning up after experiments. Toxiproxy toxics, tc rules, and injected failures persist if not removed:

# Always clean up
tc qdisc del dev eth0 root 2>/dev/null
toxiproxy-cli toxic delete --all
kubectl delete chaosengine --all -n staging

Game Day FAQ

How often should we run chaos experiments?

Start with monthly game days in staging. As confidence grows, move to weekly automated experiments in production with narrow blast radius. Netflix runs Chaos Monkey daily in production.

What metrics should we monitor during experiments?

Track these SLOs:

• Error rate (should stay < 1%)
• p99 latency (should stay within SLO)
• Throughput (should not drop > 10%)
• Recovery time (time to return to steady state)

Can chaos engineering work without Kubernetes?

Yes. Tools like Gremlin and Toxiproxy work on VMs, bare metal, and containers. AWS FIS works with EC2, ECS, and RDS. The principles are the same regardless of platform.

Performance Tips

1. Run experiments in off-peak hours. Even with narrow blast radius, experiments add load:

# Schedule for 2am when traffic is lowest
0 2 * * * /usr/local/bin/chaos-experiment.sh

1. Use short experiment durations. 30-60 seconds is enough to observe behavior:

# Litmus: 30 seconds max
env:
  - name: TOTAL_CHAOS_DURATION
    value: '30'

1. Cache vulnerability scans. If chaos experiments depend on scan results, cache them:

import functools

@functools.lru_cache(maxsize=1)
def get_service_inventory():
    return fetch_service_inventory()  # Expensive call

1. Parallelize steady-state checks. Check multiple endpoints simultaneously:

import concurrent.futures

def check_all_endpoints(urls):
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        results = list(executor.map(check_endpoint, urls))
    return all(results)