Chaos engineering

Chaos engineering is the practice of running controlled experiments that inject failures into a system, such as killing a server, adding network delay, filling a disk or making a dependency return errors, to see whether it keeps working. The idea is that in large distributed systems, failures will happen anyway; better to discover how your system reacts on a Tuesday afternoon with the team watching than at 3 a.m. during a real outage.

An experiment follows a scientific shape. First you define the steady state: a measurable sign that things are normal, like orders per minute or error rate. Then you form a hypothesis ("if one database replica dies, checkout keeps working"), inject the fault, and look for a difference. Good practice keeps the blast radius small, starting with a fraction of traffic or a test environment, and always has a stop button that rolls everything back.

Netflix popularized the idea with Chaos Monkey, a tool that randomly shut down its production servers so engineers had to build services that survive it. Today there are open source tools like Chaos Mesh and LitmusChaos for Kubernetes, and managed ones like AWS Fault Injection Service. Chaos engineering only makes sense once you have solid observability; otherwise you break things and cannot tell what happened.