Chaos Engineer Pro

A senior chaos engineering agent that designs and executes controlled failure experiments, helping you build systems that remain resilient under stress through scientific experimentation, failure injection, and systematic resilience validation.

When to Use This Agent

Choose Chaos Engineer Pro when:

• Validating system resilience before major launches or traffic events
• Designing game days and failure injection experiments
• Implementing chaos testing in CI/CD pipelines
• Investigating whether systems degrade gracefully under component failure
• Building confidence in disaster recovery and failover procedures

Consider alternatives when:

• Fixing known bugs or performance issues (use a debugging/performance agent)
• Setting up monitoring and alerting (use an observability agent)
• Writing unit or integration tests (use a test engineering agent)

Quick Start

Copy
# .claude/agents/chaos-engineer-pro.yml
name: Chaos Engineer Pro
description: Design and run resilience experiments
model: claude-sonnet
tools:
  - Read
  - Write
  - Bash
  - Glob
  - Grep

Example invocation:

claude "Design a chaos experiment to test our payment service resilience when the database becomes slow (200ms+ latency) during peak traffic"

Core Concepts

Experiment Design Framework

Phase	Action	Output
1. Hypothesis	Define expected steady state and failure behavior	Testable prediction
2. Scope	Identify blast radius and abort conditions	Safety boundaries
3. Variables	Define what to inject and what to measure	Experiment parameters
4. Execution	Run the experiment in controlled environment	Raw observations
5. Analysis	Compare results against hypothesis	Validated/invalidated findings
6. Action	Fix weaknesses or document resilience	Hardened system

Experiment Template

Copy
## Experiment: Database Latency Impact on Payment Service

### Hypothesis
When database latency increases to 200ms (from baseline 5ms),
the payment service will:
- Continue processing payments within 3-second SLA
- Queue overflow will trigger circuit breaker after 30s
- No payment data will be lost or corrupted

### Steady State
- Payment success rate: 99.5%+
- P99 latency: <2 seconds
- Queue depth: <100 messages

### Method
1. Start load generator at 100 req/s (normal traffic)
2. Inject 200ms latency on database connections using tc netem
3. Monitor for 5 minutes
4. Increase latency to 500ms
5. Monitor for 5 minutes
6. Remove latency injection
7. Verify recovery within 2 minutes

### Abort Conditions
- Payment error rate exceeds 10%
- Data corruption detected
- Cascading failure to non-payment services
- Any production customer impact (staging only)

### Measurements
- Payment success/failure rate per second
- P50/P95/P99 response latency
- Circuit breaker state transitions
- Queue depth and processing rate
- Error logs and exception counts

Failure Injection Techniques

Copy
# Network latency injection (Linux tc)
tc qdisc add dev eth0 root netem delay 200ms 50ms

# CPU stress
stress-ng --cpu 4 --cpu-load 90 --timeout 60s

# Memory pressure
stress-ng --vm 2 --vm-bytes 80% --timeout 60s

# Disk I/O saturation
stress-ng --io 4 --hdd 2 --timeout 60s

# DNS failure simulation
iptables -A OUTPUT -p udp --dport 53 -j DROP

# Service crash simulation (Kubernetes)
kubectl delete pod payment-service-abc123 --grace-period=0

# Network partition (block traffic between services)
iptables -A OUTPUT -d 10.0.1.0/24 -j DROP

Configuration

Parameter	Description	Default
environment	Target environment (staging, canary, production)	staging
blast_radius	Maximum affected scope (pod, service, zone)	pod
duration	Maximum experiment duration	5m
abort_threshold	Error rate triggering automatic abort	10%
tooling	Chaos framework (litmus, chaosmonkey, gremlin, manual)	manual
monitoring	Observability platform for measurements	Auto-detect

Best Practices

1. Always start in staging with the smallest possible blast radius. Run the first iteration of every experiment against a single pod or instance in staging. Only expand to multi-instance staging after validating that the experiment does not cause data corruption or irreversible state changes. Production chaos experiments require executive sign-off, established abort procedures, and proven staging results.

2. Define abort conditions before starting any experiment. Every experiment needs explicit, measurable criteria that trigger an immediate halt: error rate thresholds, latency ceilings, data integrity checks, and customer impact signals. Automate abort triggers where possible — a human monitoring dashboards may react too slowly. The abort procedure should be tested separately before the experiment.

3. Measure from the customer's perspective, not just internal metrics. Internal service metrics may show healthy while customers experience failures. Place synthetic transactions that mirror real user flows and measure their success rate and latency throughout the experiment. If your payment experiment shows all services healthy but customers cannot complete purchases, the system has failed regardless of internal metrics.

4. Document every experiment result, whether it confirms or invalidates the hypothesis. A confirmed hypothesis (the system behaved as expected) is valuable evidence of resilience. An invalidated hypothesis (the system failed unexpectedly) reveals a weakness to fix. Both outcomes advance system understanding. Store results in a shared location so future engineers can reference them when making architectural decisions.

5. Schedule regular game days to build organizational muscle memory. Resilience is not just technical — it requires teams that know how to respond to failures. Monthly game days with rotating incident commanders and different failure scenarios build the human reflexes needed during real incidents. Track mean time to detect, mean time to mitigate, and communication effectiveness across game days.

Common Issues

Chaos experiments cause unexpected cascading failures. An experiment targeting one service triggers failures in seemingly unrelated services due to hidden dependencies. Before running chaos experiments, map service dependencies thoroughly using distributed tracing data. Start with dependency mapping experiments (kill service X, observe which health checks fail) before injecting realistic failure scenarios. This dependency map itself is a valuable artifact.

Experiments produce inconsistent results between runs. Non-deterministic behavior in chaos experiments often comes from variable load patterns, shared infrastructure (noisy neighbors), or race conditions in the system under test. Control for load by using consistent synthetic traffic. Run experiments during known-quiet periods. Execute each experiment multiple times and look for statistical consistency rather than single-run results.

Team resistance to running chaos experiments in production. Engineers rightfully fear breaking production. Build confidence incrementally: start with read-only experiments (add latency to monitoring queries), then move to non-critical paths (recommendation service, not payment service), then critical paths during low-traffic windows. Share results widely to demonstrate that controlled experiments prevent uncontrolled production incidents.