Chaos engineering – when, where and how to use it

There may never be a more perfect time to experiment with chaos engineering than right now, in 2020, while many IT teams and their end customers continue to work remotely as Covid-19 rages on.  There’s been much written about chaos engineering, particularly its impact on DevOps organizations.

Chaos engineering (CE) entails experimenting on a system to understand the system’s capability for handling turbulent conditions in production. CE was born in the cloud age and popularized by Netflix as a best practice for building better apps and sites.  Netflix defines chaos engineering as different from testing in that its primary goal is to deliver new information about a system.

The regular application of chaos engineering methods and tools can result in more resilient systems, and resiliency has been a presiding theme this year. Yet CE isn’t just a valuable practice for developers; IT operations professionals are using chaos engineering to understand the limits of technology platforms and make adjustments to improve performance and uptime. Chaos engineering is just as relevant to monitoring systems (in the corporate data center or elsewhere) remotely as it is for monitoring and troubleshooting cloud-based services. An IT administrator working from home could use chaos engineering methods to regularly test a Jira instance in the corporate office and see the user impact of potential bottlenecks, dependencies and broken connections.

Solid use cases for chaos engineering

Knowledge transfer

Current times have revealed the limitations of the traditional IT “war room,” in that IT and engineering professionals have needed to quickly figure out new ways to collaborate and solve problems when working in isolation. CE experiments help an individual troubleshoot an issue or improve capacity planning by showing the undercover application behavior. By disrupting a key service using chaos engineering methodologies, you can get insight into how an app or system runs, performs and its dependencies. It’s akin to replacing a new spark plug in your car with one ready to be replaced and seeing the cascading effects starting with issues starting the car and accelerating. Using CE as a knowledge gathering and transfer tool, IT operations teams can get a picture of the known unknowns and unknown unknowns, such as unknown application dependencies and network spanning tree issues.  Such analysis can live in a central repository for all to see — whether that’s GitHub or an ITOM or ITSM system.

Performance

A core chaos engineering use case is stress testing IT infrastructure for load and dependencies. This can be accomplished by changing parameters for CPU, memory or disk usage and then applying automated testing to see what changes in performance. You might find that by spiking CPU usage, suddenly users have a 15% increase in web time latency. Depending upon the application’s business priority and thresholds for user experience, that result can dictate whether you will need to change configurations or add infrastructure to the resources supporting your application.

Service bottlenecks

Web applications are often composed of several interconnected micro-services. Many of these connections are not well documented, but if one service fails it can have a disastrous effect on one or more business applications. A CE experiment could place an artificial load on a service or another action to take it out of normal conditions and monitor the impact.

Getting started

There are a few things to consider when embarking on chaos engineering for IT operations:

Do we need special tools?

There are a variety of tools available now, such as those from Simian Army and Gremlin, although what’s more important is the methodology and skills.

How often should we run experiments?

It’s best to integrate chaos engineering processes with your regular release cycle, whether that is quarterly or sooner. Follow a schedule that is repeatable, regardless of use case.

How can we introduce chaos engineering in our production environment without creating unnecessary risk to our critical systems and customer experience?

CE as a practice is intended to instill more confidence for IT Operations teams and Development teams deploying to production. As a result, introducing it into production workloads is a necessary task in order to ensure that your production systems are performant, stable, and can operate efficiently even under abnormal stress. That being said, teams still hesitant to perform chaos engineering in production can:

1. Put boundaries in place to minimize blast radius in case of serious failure. (eg, only test in one pod, or one cluster, before testing others)
2. Have appropriate personnel in place to troubleshoot in case something goes wrong.
3. Test in production only if error budgets and SLOs / SLAs are already being met and some risk is acceptable.

What are some well-known best practices in running a chaos engineering experiment?

• First you’ll want to define the current steady state through a baseline analysis. Start with something relatively simple and small such as network utilization.
• Next define optimal conditions: what engineers should expect on a normal day, on an easy day and on a very hard day. The latter will be the stress test.
• Form a hypothesis as to where and how the system will break.
• Execute a real-world event by, for example, taking down a firewall that severs connectivity to an ISP.
• Validate the hypothesis. Monitor utilization and network throughput during the test and see where the system fell over. Did something entirely unexpected occur?  Stabilize, document and remediate.

As IT organizations continue to flex strategies and tools with unpredictable user needs, DevOps methods like chaos engineering will grow in usefulness. Chaos engineering is a proven best practice for IT operations teams to understand the implications of errant conditions which can threaten critical applications and websites.

This article was contributed by Michael Fisher is Group Product Manager at OpsRamp.