In this lesson, we will introduce resiliency and practices related to having our systems able to function despite experiencing problems.

Introduction to Resiliency

_The dashboard flashes red across nearly every service. The developers rush to try to locate the problem. They quickly realize that the main server, which hosts the core of the application, has had its power supply fail. There is supposed to be a backup, but the operations team hasn’t provisioned it yet. Customer complaints start rolling in as the team rushes to replace the power supply. How could this have been avoided?_

When it comes to software systems, time really is money — unexpected downtime can result in a loss of transactions, decay of customer trust, and a host of other issues. Failure will always be a part of our systems, however with the right preparation, we can build systems that are resilient to failure. 

In this lesson, we will introduce **resiliency**, a system’s ability to continue to perform despite experiencing problems. Creating a resilient system allows our services to be **highly-available**, which means our customers can access our functionality a vast majority of the time. 

To ensure resiliency, we can apply concepts of DevOps culture such as systems-level thinking, feedback, and continuous experimentation. We need to continuously monitor the whole system to understand how the components work together, learn from our system’s failures and create policies that respond to those failures.


A system dashboard in the middle of quite an emergency. With multiple issues visible at the same time, it can be difficult to understand the root cause of the problem. Times like these can often be avoided through the use of resiliency strategies.

What is Resiliency?

_Bloop Co wants to make sure that their system can handle thousands of new users attempting to buy their product. One of the engineers has created a test. The company gathers around as simulated requests flood into the system. 1000 requests, 2000 requests, 5000 requests, the system is able to handle more and more requests as they come in. It looks like some of these resiliency methods have been working!_

Remember, we want to know how our system will perform under difficult circumstances. It makes sense then to create some problems on purpose, to see how our system responds. Let's take a look at some ways engineers test the resiliency of their systems.

**Penetration testing** involves simulating cyber-attacks to try to exploit security vulnerabilities. Penetration testing gives us a chance to see how our system might respond to a malicious user. Using penetration testing allows us to identify holes in our security that we need to fix.

**Load testing** seeks to replicate situations in which the system is under heavy use. Load testing might simulate millions of customers trying to access our site all at once. Load testing can help us identify areas in which the system will break under real-world conditions.

Each of these practices can help us identify and then remove weak spots within our system. Sometimes it takes breaking a system to make it better!


A system undergoing load testing, receiving requests until it starts to smoke.

Testing Resiliency

In this illustration, waves of requests eventually take down a server.

In 2020 Google reported dealing with a [DDOS](https://www.codecademy.com/resources/blog/what-is-a-ddos-attack/) attack that lasted for over six months. At peak times the traffic created by the attackers was at 2.5 Terabytes per second sent to over 160,000 servers. This was at the time, the largest DDOS attack by traffic volume ever recorded. Events such as these showcase that malicious actors have been creating ever larger and more powerful attacks.

Cyber-attacks are attempts to disrupt system services or steal an organization's data. They can happen to businesses of different sizes and types. Some common types of cyber-attack include:

* **Distributed Denial of Service (DDOS)** attacks try to crash a target by overwhelming it with requests.
* **SQL injection** attacks try to run malicious database queries to reveal internal information.

Some techniques to handle these kinds of attacks include:

* **Filtering**: Applications might have an infrastructure layer in front of them that detects incoming DDOS attacks and ignores similar traffic.
* **Validation**: Checking that requests are valid and do not contain malicious code can help prevent attacks such as SQL injection.

When thinking about cyber-attacks, it is important to consider what attackers generally want. The prime target for most cyber-attacks is data. Cyber criminals often want to gain access to or destroy sensitive data. By employing security best practices we can be better protected against cyber-attacks.

> Learn more about cyber-attacks in our [Introduction to Cyber Security course](https://www.codecademy.com/learn/introduction-to-cybersecurity)!


Cyber Attacks

In this chart, the downtime of a system is graphed over time. There are two peaks in the chart, caused by a cloud outage and a database failure.

_Over the last few months, Bloop Co. has been trying to improve its systems' resiliency. But how do they know that they are on the right track? Which of their efforts is most successful? What are further areas they need to address? The next several exercises seek to answer these questions._

Without the ability to measure our systems’ resiliency, we can't be sure how effective our current strategy is. We can use aspects of monitoring to measure our system's responses to problems. Some important metrics that indicate a system's resiliency include:

* **Uptime**: what percentage of the time is our system available?
* **Recovery speed**: when an outage occurs, how long does it take for the system to become available again?

These metrics tell us how our system handles issues, but what do we compare them against? A pair of benchmarks that we can compare our system's performance against are recovery time objective (RTO) and recovery point objective (RPO). 

### Recovery Time Objective (RTO)

The **RTO** is the amount of time an application can be unavailable before it causes significant harm to the business. Imagine that a business has promised its users that it will never go down for more than an hour at a time. This business has set their RTO to be one hour. If the business goes down for more than that time, it will have violated an important promise to the customer.

### Recovery Point Objective RPO

The **RPO** is the acceptable amount of data loss after a system outage. Different applications have varying levels of data importance. A popular bank losing minutes of transaction data might be a nightmare. Losing hours of progress in an online multiplayer system would be unfortunate, but not a disaster. This acceptable level of data that can be lost might affect how often data is backed up, or cause adjustments to the RTO.

Benchmarks such as these can help us establish target levels of uptime and recovery time for our business. These targets can help us be aware of when our systems are approaching a critical "danger zone".


Measuring Resiliency

This gif depicts a system losing connection to a primary database and switching to a backup.

_The Data team was excited to roll out the new database system they had been working on for months. However, when the database was deployed on production hardware, something was wrong. All of the database requests were getting immediately rejected. It turned out that a configuration property was still set to the testing environment, causing the connection to fail in production._

Within our organization, internal failures will occur that are our own responsibility — and are often our own doing. Let's explore some of the ways these problems occur and how to mitigate them.

### System Changes

Updates to our system's hardware, dependencies, or code all have the potential to make our system fail or behave unexpectedly. These issues are best mitigated through a comprehensive suite of automated tests performed prior to completing any change. Change-management processes should also exist to:

* Clearly identify a change
* Determine how the change will be made
* How the change can be reviewed
* How the change can be rolled back

However, not all internal issues are our own doing. Some issues occur from our systems existing over time.

### Hardware Failures

Time passing can sometimes be enough of a change for a system to break down. Over time, hardware components, like hard drives and power supplies, reach the end of their lifespan and fail. 

**Redundancy** is one of the most common methods for providing resiliency against hardware failures. Many computer users make use of a backup hard-drive in case our primary one fails. We don't need two hard-drives, until a problem happens and we suddenly wish we had a backup.

To combat this, organizations can duplicate their hardware components. This redundancy can allow for a seamless switchover to a backup component when a failure occurs. Despite an increase in the cost and complexity of managing backup components, redundancy can go a long way in ensuring high availability of our systems.


Internal Failures

_The engineering team waits anxiously as their manager walks around in the server room. Suddenly, the manager pulls the plug on one of the server racks. The team watches the monitors. An alert goes off, but other servers come online. It looks like the system is recovering. They’ve passed a "Chaos Monkey" test for the first time._

**Chaos engineering** is the practice of performing experiments on a system in order to test its resiliency. Chaos engineering emerged as a practice in the early 2010s in Netflix, whose engineers introduced the idea of a “chaos monkey“. This chaos monkey would roam around the server room randomly disconnecting pieces of equipment. 

> There was no real monkey – these actions were performed at first by people and later through automation.

These kinds of chaotic experiments more closely replicate the random nature of real-world failure. Having a system be able to withstand chaos provides a much more powerful guarantee that it can handle “anything”.

At a much larger scale than the chaos monkey are _disaster recovery exercises_. In these exercises, companies simulate a large-scale event happening to their infrastructure. Whole rooms of servers or external dependencies might suddenly go out all at once. Companies will go through the strategy for dealing with these scenarios and see how their responses might pan out in real-time. These kinds of strategies can help companies prepare for a rare but catastrophic event.


Simulated Failure

Over the course of this lesson, we've discussed resiliency, a system's ability to perform under problematic conditions. Common conditions we need our systems to be resilient against include:

* **Internal failures**, including hardware, software, and process breakdowns
* **External failures**, including APIs, external services, and third-party providers
* **Malicious attacks**, such as DDOS and SQL injection attacks

There are a variety of practices that allow us to measure how resilient our systems are. Some common practices include:

* Having backups for our infrastructure
* Limiting our crucial dependencies
* Validating client requests

In addition to using these methods, we also need to measure how resilient our systems are. We can use metrics such as the time needed to solve a problem or the amount of downtime our system experiences. Another option is to simulate issues using techniques such as:

* **Penetration testing and load testing**
* **Chaos Engineering**
* **Disaster recovery exercises**

Resiliency practices allow us to provide critical services even under adverse conditions. Building resiliency relies on DevOps cultural practices such as continuous experimentation and learning from failure. Let's start building systems capable of weathering any storm!


Internal problems: these problems come from within the components of the system that we control. Internal problems include in-house hardware issues and software bugs.

External problems: these problems arise from dependencies we have on other parties outside of our control. External problems might include issues with an API, or a cloud service our application relies on.

Malicious actors: these problems stem from other people (or sometimes bots) that seek to disrupt or exploit our services for a variety of reasons.

Review

_At an emergency call center, something wasn’t right. All of the calls coming in were apparently coming from the same spot in the middle of the Atlantic ocean! It turned out the third-party geo-location service that the call center was using had stopped working. For each address query, the service was simply responding with latitude and longitude (0,0). This default error value made it look like the dispatch center was suddenly serving Atlantis!_

Some issues occur outside of our organization that we can only react to. Common problems involving external dependencies include:
* A dependency no longer being supported
* A dependency being taken down
* A dependency having an internal outage of its own

There are a variety of methods we can use in order to help resolve issues with external dependencies:

* We can be on the lookout for news of scheduled outages, vulnerabilities, or cancellations from our most important dependencies. 
* We can define fallback strategies for our external dependencies. If we detect that a dependency is not working, we might switch to a different dependency, or hide the functionality that depends on the failing service.
* We can automatically update dependencies as new versions are released. However, there is the possibility that a change in the new dependency version can result in our system breaking.

When using external services, we open up our system to issues that are often outside of our control. On the other hand, external services enable the development of far more powerful applications than we could create on our own. By implementing resiliency strategies, we can have the best of both worlds.


An application becoming disconnected from the cloud.

External Failures

Internal problems: these problems come from within the components of the system that we control. Internal problems include in-house hardware issues and software bugs.

External problems: these problems arise from dependencies we have on other parties outside of our control. External problems might include issues with an API, or a cloud service our application relies on.

Malicious actors: these problems stem from other people (or sometimes bots) that seek to disrupt or exploit our services for a variety of reasons.



*It's been a hectic week for Karla. First, there was a power supply failure that took down the main server. Then, on Wednesday, Karla's company lost support for the payment processing service they have been using for years. Today, the home page is receiving a suspiciously large amount of traffic that isn't actually interacting with the site — it looks like it's probably a cyber-attack. What's going to be thrown at this company next?*

Encountering problems is an intrinsic part of dealing with software systems. However, we can make significant gains in resiliency by designing our system to handle some of the most common issues it will face.

The common types of system problems fit into the following categories:

* **Internal problems**: these problems come from within the components of the system that we control. Internal problems include in-house hardware issues and software bugs.
* **External problems**: these problems arise from dependencies we have on other parties outside of our control. External problems might include issues with an API, or a cloud service our application relies on.
* **Malicious actors**: these problems stem from other people (or sometimes bots) that seek to disrupt or exploit our services for a variety of reasons.

Through managing these threats, we can make our systems more resilient!