Here is where we will introduce Monitoring and how this DevOps practice can help us gain insights into the health and performance of a system. 

Introduction to Monitoring

This image shows a character flustered by the the multitude of active alerts. 

_Callen received a text alert warning her that the company page can't connect to the database. Prometheus revealed that the database API call failed. Callen quickly switches the system to the backup database. The company’s webpage is restored and the day is saved!_

When a system experiences an issue and isn’t able to fix itself, an alert is triggered. An **alert** is a notification informing about a change of state, usually a problem. There are a variety of ways to deliver alerts:

* Tickets
* Email alerts
* Slack alerts
* SMS/phone calls

Imagine receiving alerts across all of these channels, all throughout the day. We may develop alert fatigue, where we start ignoring alerts or turning them off. To avoid alert fatigue:

* Only alert when immediate human intervention is required
* Alert based on customer facing issues
* Set clear ways to indicate urgency
* Ensure an alert is not a copy of another

With proper restraint, alerting is a critical component of system monitoring. Alerts provide context to help teams solve an issue before it becomes a crisis.


Alerting

_Klaus runs a popular retail website. He wonders if his website is meeting his customers’ expectations. If not, how does Klaus build features to further improve customer satisfaction?_

To relieve Klaus’s uncertainty, he can tie his business goals and objectives to the data he is receiving. This is where SLOs, SLIs, and SLAs can come in. 

<h6>Service Level Objective (SLO)</h6>

A **Service Level Objective** is a range of valid measurements for a metric. An SLO might define that the page response time should be less than 100 milliseconds. Measurements going outside of these ranges tend to require action. SLOs might be defined in terms of:

* Latency
* Availability
* Error rate
* And many more metrics!

A goal is only helpful if we know where we are in relation to it. While an SLO defines where we want to be, and SLI says where we are now.

<h6>Service Level Indicator (SLI)</h6>

A **Service Level Indicator** is the current measurements of a metric related to an SLO. Let's say that one of our SLOs is that response time should be less than 100 milliseconds. Our system measures that current load times are approximately 75 milliseconds. Our SLI for that SLO would be within a valid range.

SLOs and SLIs might be useful for internal performance, but what of promises we make to consumers? Our next term, the SLA, is for this purpose.

<h6>Service Level Agreement (SLA)</h6>

A **Service Level Agreement** is a contract with consumers. Not every business will have an SLA, as they carry a degree of legal responsibility. SLA binds the business to the level of expected service promised to a business’s customers. An SLA might define that a business' services be available 99.99% of the time. While breaking an SLO is an issue, breaking an SLA is a major problem. A team would want to take corrective action far before an SLA is breached.

SLOs, SLIs and SLAs allow businesses to tie promises and goals to the data coming from their systems. Using these terms make it much more clear when a system metric is jeopardizing the health of a business.



This image shows a table summarizing the SLX term and its definition. 

SLOs, SLAs, and SLIs

_Clark added a new service to handle more traffic for his growing business. But he doesn’t know how his new application is doing! What information could help him gain insight into his system?_

Clark can measure how his application is operating with the help of metrics. **Metrics** express a value relevant to the system at a specific point of time. Here are some key metrics for monitoring system health and performance:

<details>
<summary>
Latency
</summary>
Latency is the time between the start of an event, such as serving a request, to its completion. This metric is a key indicator of performance. 
</details>

<br/>

<details>
<summary>
Traffic/Connections
</summary>
Traffic is the amount of system usage over time. An abnormal amount of traffic can require scaling to maintain performance.
</details>

<br/>

<details>
<summary>
Errors
</summary>
An error is an invalid state our system has reached. Examples include exceeding a memory limit or reading a corrupted data file. The rate of errors returned by a service can indicate deeper issues.
</details>

<br/>

<details>
<summary>
Saturation
</summary>
Saturation describes the load on our system’s resources. Our system reaching its limits can result in poor performance.
</details>
<br/>

Tracking these metrics can give teams a broad view of system health and help diagnose issues. 


This image shows a spread of graphs to represent the metrics garnered from monitoring. 

Metrics

Congratulations! We have learned so much about monitoring systems. Let's review what we've covered: 

* Monitoring is the practice of collecting metrics to gain insights on our systems.
* Insights into our systems tell us about its health and problems.
* We can tie our business objectives to metrics with SLAs, SLOs, and SLIs.
* Alerts should be sent out for customer facing issues requiring immediate attention.
* We can use monitoring tools to aggregate and visualize our system insights.
* Observability is the degree to which our monitoring helps us solve system problems.
* A high-quality monitoring system allows us to better diagnose and resolve system issues!

Monitoring is critical to applying DevOps in any organization. Start gaining system insights with monitoring today!


Image shows person staring at a display of charts, graphs, and other data. 

Review

This image shows a laptop with a pop out window showing data, graphs, and charts. 

_Lenny runs an online business selling refurbished furniture. He's nervous about his website crashing during peak traffic periods. What tools can Lenny use to keep informed about his website's status?_

There are several monitoring tools that can aid in understanding Lenny’s infrastructure. Teams practicing DevOps use monitoring tools to:

* Oversee resources such as application services, databases, and other processes
* View system health
* Capture real-time health statuses and performance metrics. 

In many situations, monitoring is achieved using a combination of tools. Big picture trends, like performance, can be monitored by a tool like Prometheus. Meanwhile, a finer piece of detail, such as the time taken for a database query, may be monitored using Monyog. When Lenny uses these kinds of combinations he can identify both the problem and its cause.

There are several popular monitoring tools available to measure a variety of metrics. Let's take a brief look at monitoring tools on the market:

<details>
<summary>
Zabbix
</summary>
Zabbix offers many features such as a centralized, easy-to-use web interface. Zabbix can directly monitor Java applications and offers built-in graphing and visualization capabilities.
</details>

<br/>

<details>
<summary>
Prometheus
</summary>
Prometheus collects metrics values from target systems. This monitoring tool uses PromQL, a query language, that lets users select data in real time.
</details>

<br/>

<details>
<summary>
Sensu
</summary>
Sensu is a highly extensible and scalable system that monitors cloud infrastructures. Using Sensu, the monitoring requirements can be implemented as code.
</details>

<br/>

<details>
<summary>
Monyog
</summary>
Monyog is used by database teams to detect issues affecting MySQL database performance.
</details>
<br/>

Monitoring tools offer a clearer picture of how applications and infrastructure are working. Teams can use these tools to assess system health and track down problems.


Monitoring Tools

_Over time, Bloop Co's IT infrastructure has become very complex. Jeff's team used to be able to inspect the health of each component, but those days are long gone. When the system slows or an outage occurs, Jeff and his team don't know where to begin. How can they check system health as the company’s infrastructure expands?_

This system needs monitoring.

**Monitoring** helps teams understand the state of their systems based on gathered data. The monitoring stage of the DevOps lifecycle is crucial in gathering real-time insights on:

* Performance
* Health status
* Scalability

In the exercises ahead, we will explore the following key topics:

* Metrics utilized to assess the health of the system
* Comparing metrics to business objectives 
* Monitoring tools 
* Effective alerting
* Observability
* Monitoring quality

Learning about these topics will deepen our understanding about our complex systems. Let's jump right in!



This image shows the diagram of DevOps lifecycle. 

What is Monitoring

_A modern pilot relies heavily on their plane's equipment. These allow navigation in even the most stormy weather. For a complex software system, we need a high quality monitoring system to make sense of it all. What are some ways we can make sure our instrumentation is able to navigate any crisis?_

For system monitoring to be effective, it takes a significant effort. Let’s take a look at some practices to improve monitoring quality: 

- Define actionable alerts that are customized to the needs of our organization
- Collect application logs and make this data available and understandable.
- Incorporate logging into the build and deployment process.
- Define custom, actionable alerts that are relevant to the organization. 

Another key indicator of monitoring quality is the way our organization handles alerts. Be on the lookout for:

<details><summary>False negatives</summary>
Pay attention when a user-affecting issue has happened, and the system does not alert us. The lack of an alert indicates a hole in our monitoring.  We should hold a retrospective meeting to find out what metrics could have alerted us to the problem.
</details>

<br/>

<details>
<summary>
False positives
</summary>
This occurs when an alert is generated, but there is nothing wrong with the system. The threshold for an alert may need to be adjusted, or the alert might need to be deleted altogether.
</details>
 
<br/>

<details>
<summary>
Unactionable alerts
</summary>
This type of alert has little to do with a problem and doesn’t need anything done. Like false negatives, we should reduce or delete unactionable alerts.
</details>

<br/>

Useless or incorrect alerts add to the chance that valuable alerts will be ignored or unseen. Keeping our alerts at a high quality ensures that each is given proper attention.

By following these best practices, we are off to a great start with monitoring quality!


This image shows an image of airplane cockpit. 

Monitoring Quality

_Caleb received a Slack alert about an issue with one of the company’s backend applications. How does Caleb and his team track down the error in a complex maze of backend services?_

Let's take a look at the general steps Caleb’s team might go through when an issue arises:

1) Evaluate usage and performance data
<br/>
2) Identify the cause of the issue. 
<br/>
3) Apply the appropriate solution, restoring system performance.

**Observability** is the degree to which a system's information can be used to locate and fix a problem. In a system with high observability, a team can more easily trace, diagnose, and fix the problem. With poor observability, the data does little to help.

To improve a system’s observability:

* Make sure team is aligned with service level objectives 
* Create meaningful alerts
* Optimize application logging by ensuring messages are informational and descriptive
* Automate work processes

Maintaining an observable system enables teams to proactively monitor and track for errors. 


This image shows a laptop with an error. The error expands into web of nodes. 