Reliability & Chaos Testing

What it is

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions. You form a hypothesis about steady state, inject a realistic fault, and observe whether the system holds.

1
  steady state ──► inject fault ──► observe ──► learn
2
      │                 │              │           │
3
   p99, errors      kill pod       did SLO hold?  fix / accept
4
                    add latency
5
                    partition net

Related practice: reliability testing keeps the focus on the SLOs you promised (availability, error budget) rather than on a single outage.

When to use it / when to avoid it

Use it for

Systems with replication, retries, failover or autoscaling claims to verify
Game days: a scheduled, controlled experiment with a rollback ready
Validating that timeouts, circuit breakers and retries actually work
Finding the failure a real datacentre or network event will eventually cause

Avoid or reduce it when

You have no observability — you cannot learn from an experiment you cannot see
There is no blast-radius control (feature flags, canary, abort button)
You start in production before proving the experiment in staging
Nobody owns the SLOs; without a target, “resilient” is undefined

Frameworks & tools

Tool	Ecosystem	Notes
Chaos Mesh	Kubernetes	CRDs for pod, network, IO, DNS and time chaos
Litmus	Kubernetes	ChaosHub library, experiment workflows
Gremlin	SaaS / agent	Managed attacks, safety controls
Toxiproxy	Multi	TCP proxy to inject latency and resets
AWS Fault Injection Simulator	AWS	Managed cloud-level faults
Pumba	Docker	Chaos for plain Docker containers
Chaos Monkey	JVM / Spinnaker	The original random-termination tool
kube-monkey	Kubernetes	Schedules random pod kills

Quick setup

1
docker run -d --name toxiproxy -p 8474:8474 -p 8666:8666 \
2
  ghcr.io/shopify/toxiproxy:2.9.0
3
toxiproxy-cli create postgres --listen 0.0.0.0:8666 --upstream db:5432
4
toxiproxy-cli toxic add postgres -t latency -a latency=300

1
helm repo add chaos-mesh https://charts.chaos-mesh.org
2
helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --create-namespace

1
npm install -D toxiproxy-node-client

1
pip install toxiproxy-python

1
go get github.com/Shopify/toxiproxy/v2/client

1
<!-- Testcontainers Toxiproxy module -->
2
<dependency>
3
  <groupId>org.testcontainers</groupId>
4
  <artifactId>toxiproxy</artifactId>
5
  <version>1.20.1</version>
6
  <scope>test</scope>
7
</dependency>

1
dotnet add package Toxiproxy.Net

Example

1
toxiproxy-cli create postgres --listen 0.0.0.0:8666 --upstream db:5432
2
toxiproxy-cli toxic add postgres -t latency -a latency=500
3
# now run your suite; assert timeouts and circuit breakers behave
4
toxiproxy-cli toxic remove postgres -n latency_downstream

1
apiVersion: chaos-mesh.org/v1alpha1
2
kind: PodChaos
3
metadata:
4
  name: kill-api
5
spec:
6
  action: pod-kill
7
  mode: one
8
  selector:
9
    labelSelectors:
10
      app: api
11
  duration: '1m'

1
const proxy = await api.createProxy({ name: 'redis', listen: '0.0.0.0:8667', upstream: 'redis:6379' });
2
await proxy.addToxic({ type: 'latency', attributes: { latency: 200 } });

1
from toxiproxy import Toxiproxy
2

3
api = Toxiproxy()
4
proxy = api.create(name="redis", listen="0.0.0.0:8667", upstream="redis:6379")
5
proxy.add_toxic(type="latency", attributes={"latency": 200})
6
# assert the cache layer degrades gracefully, then clean up
7
proxy.destroy()

1
// wrap the HTTP client with a timeout to emulate a slow dependency
2
client := &http.Client{Timeout: 250 * time.Millisecond}
3
_, err := client.Get("http://downstream/health")
4
if err == nil {
5
  t.Fatal("expected a timeout under injected latency")
6
}

1
var proxy = new ToxiproxyContainer(DockerImageName.parse("ghcr.io/shopify/toxiproxy:2.9.0"));
2
proxy.start();
3
var toxi = proxy.getProxy(proxy.getContainerIpAddress(), 8666);
4
toxi.toxics().latency("pg-latency", ToxicDirection.DOWNSTREAM, 300);

1
var proxy = await ToxiproxyClient.CreateProxy("postgres", "0.0.0.0:8666", "db:5432");
2
await proxy.Add(new LatencyToxic { Latency = 300 });

Anti-patterns & best practices

Define steady state first. Without an SLO you cannot tell an experiment from an outage.
Constrain the blast radius. One service, one zone, an abort switch, and a feature flag around anything risky.
Run game days on a schedule. Practice drills beat a heroic scramble during a real incident.
Inject realistic faults. Real incidents are latency, partial failure and DNS — not just a clean pod kill.
Automate in staging, explore in production. Same experiment, staged confidence.
Blame the system, not the person. Every surprise is a missing guard rail; turn it into a test or a dashboard panel.