What Is Chaos Engineering?

Every distributed system fails eventually. A node dies, a network link drops packets, a dependency times out, a disk fills, a CPU pegs at 100%. The only question is whether you find out how your system responds on a random Tuesday at 3 a.m. — or on purpose, on a Wednesday afternoon, with the whole team watching and a rollback ready. Chaos engineering is choosing the second option. As the Principles of Chaos Engineering put it, it is 'the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.'

The key word is confidence. You cannot reason your way to certainty that a system of dozens of interacting services will behave under failure — the interactions are too complex to hold in one head. So instead of arguing about it in a design review, you run an experiment and let the system tell you. The discipline was born at Netflix around 2010, when the company moved to the cloud and needed every engineer to assume that any instance could vanish at any moment. Their answer was Chaos Monkey, a tool that randomly terminates production instances during business hours so that teams are forced to build services that survive the loss of any single node.

What separates chaos engineering from simply unplugging a server is that it is structured as a real experiment. The Principles of Chaos describe four steps, and they map one-to-one onto the scientific method you learned in school.

1. Define the steady state. Start by picking a measurable output that indicates the system is behaving normally — requests per second successfully served, the 99th-percentile latency, the checkout conversion rate. This is your dependent variable. Crucially, the steady state is defined in terms of what users experience, not internal metrics like CPU usage. A server running hot is not itself a problem; a customer who cannot check out is.

2. Hypothesize that the steady state holds. Form the hypothesis that this normal behavior will continue in both a control group (untouched) and an experimental group (about to be attacked). You are predicting that the failure you are about to inject will make no meaningful difference — because your redundancy, retries, and failovers should absorb it.

3. Inject real-world failures. Introduce variables that reflect events that actually happen: terminate instances, sever a network connection, spike latency, drop packets, fail a disk, exhaust CPU or memory. The emphasis on real matters — you simulate the failures that genuinely occur in production, not contrived ones, because those are the ones your users will hit.

4. Try to disprove the hypothesis. Now compare the steady state of the experimental group against the control. If they still match, your confidence in the system's resilience goes up. If they diverge — errors climb in the group you attacked while the control stays healthy — you have found a weakness. That difference is the whole point: a discovered gap you can fix on your own schedule, rather than a surprise outage on the system's schedule. As the principles note, the harder it is to disrupt the steady state, the more confidence you have.

The advanced principles add the discipline that keeps an experiment from becoming an incident. Start small. Run your first experiments in staging, with a tiny scope, and graduate toward production only as your confidence and tooling mature. Minimize the blast radius. Contain each experiment to the smallest set of traffic that can still teach you something — a single availability zone, one percent of requests, a shadow cohort — so that if the hypothesis is wrong, the damage is bounded. Run in production, carefully. The value of chaos engineering comes from testing the real system with real traffic, because staging never fully mirrors production; but you do that only with guardrails in place. Automate and run continuously, so resilience is verified on every change rather than in a one-off game day. And above all, have an abort switch: the ability to halt the experiment and roll back instantly the moment real user harm appears.

The classic experiments map directly onto the ways production actually fails: kill an instance to test redundancy, sever the network to test failover and partition tolerance, spike latency to test timeouts and retries, fail a disk to test storage failover, and exhaust CPU or memory to test back-pressure and graceful degradation. As tooling, Chaos Monkey remains the canonical starting point; commercial platforms like Gremlin provide a catalog of attacks with built-in blast-radius controls; and CNCF projects such as LitmusChaos and Chaos Mesh let Kubernetes teams declare fault-injection experiments as manifests. The discipline matters more than any single tool.

Chaos engineering does not stand alone. It is downstream of observability: you cannot detect a change in steady state you cannot measure, so good telemetry — metrics, logs, and traces — is a prerequisite, not a nice-to-have. Run an experiment without it and you are breaking things blind. It is also upstream of incident response. Every weakness you surface in a controlled experiment is one that does not become a 3 a.m. page, and the ones that do teach you something belong in your review process. Pair chaos work with a blameless incident postmortem so that findings turn into fixes, and track whether that discipline is paying off in your mean time to recovery — the faster you recover from the failures you rehearsed, the more the practice is working.

Notice how much of the definition insists on the user. The steady state is a user-facing indicator; the abort condition is real user harm; the whole point is to withstand conditions your customers experience. Resilience is not really about whether a server stays up — it is about whether the person clicking 'buy' still succeeds when a server does not. That is the gap dashboards miss: a graph can show a latency spike absorbed cleanly at the infrastructure layer while, three hops away, a specific flow quietly breaks for a subset of users.

This is where capturing real user sessions during an experiment earns its keep. When you inject a failure, an aggregate metric tells you the average held; a captured session tells you what a single real person actually saw — the spinner that never resolved, the retry that silently dropped their cart, the error toast that fired only on slow connections. BugMojo records the browser-side reality — an rrweb session replay plus the console and network state — so that the user-facing impact of a chaos experiment is visible, not inferred. The metric proves the steady state; the session shows you the human behind it. Both matter, and only one of them can be argued with in a review.