Learn how to use NumPy for statistical calculations.

Introduction to Statistics with NumPy

Another key metric that we can use in data analysis is the *median*. The median is the middle value of a dataset that’s been ordered in terms of magnitude (from lowest to highest). 

Let's look at the following array:

```py
np.array( [1, 1, 2, 3, 4, 5, 5])
```
In this example, the median would be **3**, because it is positioned half-way between the minimum value and the maximum value. 

If the length of our dataset was an even number, the median would be the value halfway between the two central values. So in the following example, the median would be **3.5**: 

```py
np.array( [1, 1, 2, 3, 4, 5, 5, 6])
```


But what if we had a very large dataset? It would get very tedious to count all of the values. Luckily, NumPy also has a function to calculate the median, `np.median`: 

```py
>>> my_array = np.array([50, 38, 291, 59, 14])
>>> np.median(my_array)
50.0
``` 





NumPy and Median

As we can see, the mean is a helpful way to quickly understand different parts of our data. However, the mean is highly influenced by the specific values in our data set. What happens when one of those values is significantly different from the rest?

Values that don’t fit within the majority of a dataset are known as *outliers*. It’s important to identify outliers because if they go unnoticed, they can skew our data and lead to error in our analysis (like determining the mean). They can also be useful in pointing out errors in our data collection. 

When we're able to identify outliers, we can then determine if they were due to an error in sample collection or whether or not they represent a significant but real deviation from the mean. 

Suppose we want to determine the average height for 3rd graders. We measure several students at the local school, but accidentally measure one student in centimeters rather than in inches. If we're not paying attention, our dataset could end up looking like this:

```py
[50, 50, 51, 49, 48, 127]
```
In this case, 127 would be an outlier.

Some outliers aren’t the result of a mistake.  For instance, suppose that one of our 3rd graders had skipped a grade and was actually a year younger than everyone else in the class:

```py
[50, 50, 51, 49, 48, 45]
```

She might be significantly shorter at 45", but her height would still be an outlier.  


Suppose that another student was just unusually tall for his age: 
```py
[50, 50, 51, 49, 48, 58.5]
```
His height of 58.5" would also be an outlier.

Outliers

As we saw in the last exercise, knowing the standard deviation of a dataset can help us understand how spread out our dataset is. 

We can find the standard deviation of a dataset using the Numpy function `np.std`:

```py
>>> nums = np.array([65, 36, 52, 91, 63, 79])
>>> np.std(nums)
17.716909687891082
```


NumPy and Standard Deviation, Part II

If we have a two-dimensional array, `np.mean` can calculate the means of the larger array as well as the interior values. 

Let's imagine a game of ring toss at a carnival. In this game, you have three different chances to get all three rings onto a stick. In our `ring_toss` array, each interior array (the arrays within the larger array) is one try, and each number is one ring toss. **1** represents a successful toss, **0** represents a fail.   

First, we can use `np.mean` to find the mean across all the arrays:

```py
>>> ring_toss = np.array([[1, 0, 0], 
                          [0, 0, 1], 
                          [1, 0, 1]])
>>> np.mean(ring_toss)
0.44444444444444442
```
To find the means of each interior array, we specify axis 1 (the "rows"):

```py
>>> np.mean(ring_toss, axis=1)
array([ 0.33333333,  0.33333333,  0.66666667])
```


To find the means of each index position (i.e, mean of all 1st tosses, mean of all 2nd tosses, ...), we specify axis 0 (the "columns"):

```py
>>> np.mean(ring_toss, axis=0)
array([ 0.66666667,  0.        ,  0.66666667])
```


Calculating the Mean of 2D Arrays

As we know, the median is the middle of a dataset: it is the number for which 50% of the samples are below, and 50% of the samples are above. But what if we wanted to find a point at which 40% of the samples are below, and 60% of the samples are above?

This type of point is called a *percentile*. The *Nth percentile* is defined as the point N% of samples lie below it.  So the point where 40% of samples are below is called the 40th percentile. Percentiles are useful measurements because they can tell us where a particular value is situated within the greater dataset. 

Let's look at the following array:
```py
d = [1, 2, 3, 4, 4, 4, 6, 6, 7, 8, 8]
```

There are 11 numbers in the dataset.  The 40th percentile will have 40% of the 10 remaining numbers below it (40% of 10 is 4) and 60% of the numbers above it (60% of 10 is 6).  So in this example, the 40th percentile is 4.

![percentile](https://content.codecademy.com/courses/numpy/NumPy%2040%20percentile.svg)

In NumPy, we can calculate percentiles using the function `np.percentile`, which takes two arguments: the **array** and the **percentile** to calculate.

Here's how we would use NumPy to calculate the 40th percentile of array `d`:

```py
>>> d = np.array([1, 2, 3, 4, 4, 4, 6, 6, 7,  8, 8])
>>> np.percentile(d, 40)
4.00
```


Percentiles, Part I

In a dataset, the median value can provide an important comparison to the mean. Unlike a mean, the median is not affected by outliers. This becomes important in *skewed* datasets, datasets whose values are not distributed evenly. Let's write a program that explores this idea further. 

Mean vs. Median

Some percentiles have specific names:
- The **25th percentile** is called the *first quartile*
- The **50th percentile** is called the *median*
- The **75th percentile** is called the *third quartile*

The minimum, first quartile, median, third quartile, and maximum of a dataset are called a *five-number summary*. This set of numbers is a great thing to compute when we get a new dataset.

The difference between the first and third quartile is a value called the *interquartile range*.  For example, say we have the following array:

```py
d = [1, 2, 3, 4, 4, 4, 6, 6, 7, 8, 8]
```

We can calculate the 25th and 75th percentiles using `np.percentile`:

```py
np.percentile(d, 25)
>>> 3.5
np.percentile(d, 75)
>>> 6.5
```

Then to find the interquartile range, we subtract the value of the 25th percentile from the value of the 75th:

```
6.5 - 3.5 = 3
```

**50% of the dataset** will lie within the interquartile range. The interquartile range gives us an idea of how spread out our data is. The smaller the interquartile range value, the less variance in our dataset. The greater the value, the larger the variance. 

Percentiles, Part II

The first statistical concept we'll explore is *mean*, also commonly referred to as an average. The mean is a useful measurement to get the center of a dataset. NumPy has a built-in function to calculate the average or mean of arrays: `np.mean`

Let's say we want to find the average number of pounds of produce a person purchases per week. We administered a survey and received 1,000 responses:

```py
survey_responses = [5, 10.2, 4, .3 ... 6.6]
```

We can then transform the dataset into a NumPy array and use the function `np.mean` to calculate the average:

```py
>>> survey_array = np.array(survey_responses)
>>> np.mean(survey_array)
5.220
```



NumPy and Mean

While the mean and median can tell us about the center of our data, they do not reflect the range of the data. That's where *standard deviation* comes in. 

Similar to the interquartile range, the standard deviation tells us the spread of the data. The larger the standard deviation, the more spread out our data is from the center. The smaller the standard deviation, the more the data is clustered around the mean. 

NumPy and Standard Deviation, Part I

We can also use `np.mean` to calculate the percent of array elements that have a certain property. 

As we know, a logical operator will evaluate each item in an array to see if it matches the specified condition. If the item matches the given condition, the item will evaluate as `True` and equal 1. If it does not match, it will be `False` and equal 0. 

When `np.mean` calculates a logical statement, the resulting mean value will be equivalent to the total number of `True` items divided by the total array length.

In our produce survey example, we can use this calculation to find out the percentage of people who bought more than 8 pounds of produce each week:

```py
>>> np.mean(survey_array > 8)
0.2
```

The logical statement `survey_array > 8` evaluates which survey answers were greater than 8, and assigns them a value of 1.  `np.mean` adds all of the 1s up and divides them by the length of `survey_array`. The resulting output tells us that 20% of responders purchased more than 8 pounds of produce. 

Mean and Logical Operations

Let's review! In this lesson, you learned how to use NumPy to analyze single-variable datasets. Here's what we covered:

- Using the `np.sort` method to locate outliers.
- Calculating central positions of a dataset using `np.mean` and `np.median`.
- Understanding the spread of our data using **percentiles** and the **interquartile range**. 
- Finding the standard deviation of a dataset using `np.std`.

In our next lesson, we'll continue our exploration of NumPy and see how we can use it to analyze different statistical distributions. Follow the checkpoints below to practice what you just learned! 

Review

One way to quickly identify outliers is by sorting our data, Once our data is sorted, we can quickly glance at the beginning or end of an array to see if some values lie far beyond the expected range.  We can use the NumPy function `np.sort` to sort our data. 

Let’s go back to our 3rd grade height example, and imagine an 8th grader walked into our experiement:

```py
>>> heights = np.array([49.7, 46.9, 62, 47.2, 47, 48.3, 48.7])
```

If we use `np.sort`, we can immediately identify the taller student since their height (62") is  noticeably outside the range of the dataset:

```py
>>> np.sort(heights)
array([ 46.9,  47. ,  47.2,  48.3,  48.7,  49.7,  62])
```




Sorting and Outliers

You're a citizen scientist who has started collecting data about rising water in the river next to where you live. For months, you painstakingly measure the water levels and enter your findings into a notebook. But at the end of it, what exactly do you have? What can all this data tell us? 

In this lesson, we'll explore how we can use NumPy to analyze data. We'll learn different methods to calculate common statistical properties of a dataset, such as finding the mean and standard deviation. By the end, you'll be able to do basic analysis of a dataset and understand how we can use statistics to come to conclusions about data. 

The statistical concepts that we'll cover include:

- Mean
- Median
- Percentiles
- Interquartile Range
- Outliers
- Standard Deviation

To start, we'll be analyzing  *single-variable* datasets. One way to think of a single-variable dataset is that it contains answers to a question. For instance, we might ask 100 people, “How tall are you?” Their heights in inches would form our dataset.

For our purposes, we'll be organizing our datasets into NumPy arrays. To learn more about NumPy arrays, take our course <a href = "https://www.codecademy.com/learn/learn-numpy-introduction" target="_blank" rel="noopener noreferrer">Learn NumPy: Introduction</a>. 