This lesson covers the different types of hypothesis tests and the situations they are most appropriate for.

Hypothesis Testing with R

In the last exercise, you saw that the probability of making a Type I error got dangerously high as you performed more t-tests.

When comparing more than two numerical datasets, the best way to preserve a Type I error probability of `0.05` is to use ANOVA. _ANOVA (Analysis of Variance)_ tests the null hypothesis that all of the datasets you are considering have the same mean. If you reject the null hypothesis with ANOVA, you're saying that at least one of the sets has a different mean; however, it does not tell you which datasets are different.

You can use the `stats` package function `aov()` to perform ANOVA on multiple datasets. `aov()` takes the different datasets combined into a data frame as an argument. For example, if you were comparing scores on a video game between math majors, writing majors, and psychology majors, you could format the data in a data frame `df_scores` as follows:

|group|score|
|-----|-----|
|math major|88|
|math major|81|
|writing major|92|
|writing major|80|
|psychology major|94|
|psychology major|83|

You can then run an ANOVA test with this line:

```r
results <- aov(score ~ group, data = df_scores)
```
Note: `score ~ group` indicates the relationship you want to analyze (i.e. how each `group`, or major, relates to `score` on the video game)

To retrieve the p-value from the results of calling `aov()`, use the `summary()` function:

```r
summary(results)
```

The null hypothesis, in this case, is that all three populations have the same mean score on this video game. If you reject this null hypothesis (if the p-value is less than `0.05`), you can say you are reasonably confident that a pair of datasets is significantly different. After using only ANOVA, however, you can't make any conclusions on which two populations have a significant difference.

Let's look at an example of ANOVA in action.

ANOVA

Suppose that you own a chain of stores that sell ants, called VeryAnts. There are three different locations: A, B, and C. You want to know if the average ant sales over the past year are significantly different between the three locations.

At first, it seems that you could perform T-tests between each pair of stores.

You know that the p-value is the probability that you incorrectly reject the null hypothesis on each t-test. The more t-tests you perform, the more likely that you are to get a false positive, a Type I error. 

For a p-value of `0.05`, if the null hypothesis is true, then the probability of obtaining a significant result is `1 – 0.05` = `0.95`. When you run another t-test, the probability of still getting a correct result is `0.95` * `0.95`, or `0.9025`. That means your probability of making an error is now close to `10%`! This error probability only gets bigger with the more t-tests you do.



Dangers of Multiple T-Tests

When using automated processes to make decisions, you need to be aware of how this automation can lead to mistakes. Computer programs can be as fallible as the humans who design them. Because of this, there is a responsibility to understand what can go wrong and what can be done to contain these foreseeable problems.

In statistical hypothesis testing, there are two types of error. A _Type I error_ occurs when a hypothesis test finds a correlation between things that are not related. This error is sometimes called a "false positive" and occurs when the null hypothesis is rejected even though it is true.

For example, consider the history and chemistry major experiment from the previous exercise. Say you run a hypothesis test on the sample data you collected and conclude that there is a significant difference in interest in volleyball between history and chemistry majors. You have rejected the null hypothesis that there is no difference between the two populations of students. If, in reality, your results were due to the groups you happened to pick (sampling error), and there actually is no significant difference in interest in volleyball between history and chemistry majors in the greater population, you have become the victim of a false positive, or a Type I error.

The second kind of error, a _Type II error_, is failing to find a correlation between things that are actually related. This error is referred to as a "false negative" and occurs when the null hypothesis is not rejected even though it is false.

For example, with the history and chemistry student experiment, say that after you perform the hypothesis test, you conclude that there is no significant difference in interest in volleyball between history and chemistry majors. You did _not_ reject the null hypothesis. If there actually is a difference in the populations as a whole, and there is a significant difference in interest in volleyball between history and chemistry majors, your test has resulted in a false negative, or a Type II error.


Type I and Type II Errors

While a hypothesis test will return a p-value indicating a level of confidence in the null hypothesis, it does not definitively claim whether you should reject the null hypothesis. To make this decision, you need to determine a threshold p-value for which all p-values below it will result in rejecting the null hypothesis. This threshold is known as the _significance level_.

A higher significance level is more likely to give a false positive, as it makes it "easier" to state that there is a difference in the populations of your data when such a difference might not actually exist. If you want to be very sure that the result is not due to sampling error, you should select a very small significance level.

It is important to choose the significance level before you perform a statistical hypothesis test. If you wait until after you receive a p-value from a test, you might pick a significance level such that you get the result you want to see. For instance, if someone is trying to publish the results of their scientific study in a journal, they might set a higher significance level that makes their results appear statistically significant. Choosing a significance level in advance helps keep everyone honest.

It is an industry-standard to set a significance level of `0.05` or less, meaning that there is a `5%` or less chance that your result is due to sampling error.

Significance Level

You begin the statistical hypothesis testing process by defining a _hypothesis_, or an assumption about your population that you want to test. A hypothesis can be written in words, but can also be explained in terms of the sample and population means you just learned about.

Say you are developing a website and want to compare the time spent on different versions of a homepage. You could run a hypothesis test to see if version A or B makes users stay on the page significantly longer. Your hypothesis might be:

`"The average time spent on homepage A is greater than the average time spent on homepage B."`

While this is a fine hypothesis to make, data analysts are often very hesitant people. They don't like to make bold claims without having data to back them up! Thus when constructing hypotheses for a hypothesis test, you want to formulate a null hypothesis. A _null hypothesis_ states that there is no difference between the populations you are comparing, and it implies that any difference seen in the sample data is due to sampling error. A null hypothesis for the same scenario is as follows:

`"The average time spent on homepage A is the same as the average time spent on homepage B."`

You could also restate this in terms of population mean:

`"The population mean of time spent on homepage A is the same as the population mean of time spent on homepage B."`

After collecting some sample data on how users interact with each homepage, you can then run a hypothesis test using the data collected to determine whether your null hypothesis is true or false, or can be rejected (i.e. there is a difference in time spent on homepage A or B).

Hypothesis Formulation

Consider the fictional business BuyPie, which sends ingredients for pies to your household so that you can make them from scratch. Suppose that a product manager hypothesizes the average age of visitors to BuyPie.com is `30`. In the past hour, the website had `100` visitors and the average age was `31`. Are the visitors older than expected? Or is this just the result of chance (sampling error) and a small sample size? 

You can test this using a One Sample T-Test. A _One Sample T-Test_ compares a sample mean to a hypothetical population mean. It answers the question "What is the probability that the sample came from a distribution with the desired mean?"

The first step is formulating a null hypothesis, which again is the hypothesis that there is no difference between the populations you are comparing. The second population in a One Sample T-Test is the hypothetical population you choose. The null hypothesis that this test examines can be phrased as follows: `"The set of samples belongs to a population with the target mean".` 

One result of a One Sample T-Test will be a _p-value_, which tells you whether or not you can reject this null hypothesis. If the p-value you receive is less than your significance level, normally `0.05`, you can reject the null hypothesis and state that there is a significant difference.

R has a function called `t.test()` in the `stats` package which can perform a One Sample T-Test for you.

`t.test()` requires two arguments, a distribution of values and an expected mean:

```r
results <- t.test(sample_distribution, mu = expected_mean)
```
* `sample_distribution` is the sample of values that were collected
* `mu` is an argument indicating the desired mean of the hypothetical population
* `expected_mean` is the value of the desired mean

`t.test()` will return, among other information we will not cover here, a p-value &mdash; this tells you how confident you can be that the sample of values came from a distribution with the specified mean.

P-values give you an idea of how confident you can be in a result. Just because you don’t have enough data to detect a difference doesn’t mean that there isn’t one. Generally, the more samples you have, the smaller a difference you can detect. 


One Sample T-Test

Suppose that last week, the average amount of time spent per visitor to a website was `25` minutes. This week, the average amount of time spent per visitor to a website was `29` minutes. Did the average time spent per visitor change (i.e. was there a statistically significant bump in user time on the site)? Or is this just part of natural fluctuations?

One way of testing whether this difference is significant is by using a Two Sample T-Test. A _Two Sample T-Test_ compares two sets of data, which are both approximately normally distributed. 

The null hypothesis, in this case, is that the two distributions have the same mean.

You can use R's `t.test()` function to perform a Two Sample T-Test, as shown below:

```r
results <- t.test(distribution_1, distribution_2)
```

When performing a Two Sample T-Test, `t.test()` takes two distributions as arguments and returns, among other information, a p-value. Remember, the p-value let's you know the probability that the difference in the means happened by chance (sampling error).

Two Sample T-Test

Phew! Nobody said hypothesis testing is easy, but you made it to the end of the lesson. Congratulations! The world of hypothesis testing is vast. There is much more you can learn, and so many applications where you can use them.

Let's review what you've learned in this lesson:
* _Samples_ are subsets of an entire _population_, and the _sample mean_ can be used to approximate the _population mean_
* The _null hypothesis_ is an assumption that there is no difference between the populations you are comparing in a hypothesis test
* _Type I Errors_ occur when a hypothesis test finds a correlation between things that are not related, and _Type II Errors_ occur when a hypothesis test fails to find a correlation between things that are actually related
* _P-Values_ indicate the probability that, assuming the null hypothesis is true, such differences in the samples you are comparing would exist
* The _Significance Level_ is a threshold p-value for which all p-values below it will result in rejecting the null hypothesis
* _One Sample T-Tests_ indicate whether a dataset belongs to a distribution with a given mean 
* _Two Sample T-Tests_ indicate whether there is a significant difference between two datasets 
* _ANOVA (Analysis of Variance)_ allows you to detect if there is a significant difference between one of multiple datasets

Review

Say you work for a major social media website. Your boss comes to you with two questions:
* does the demographic of users on your site match the company's expectation?
* did the new interface update affect user engagement?

With terabytes of user data at your hands, you decide the best way to answer these questions is with statistical hypothesis tests!

_Statistical hypothesis testing_ is a process that allows you to evaluate if a change or difference seen in a dataset is "real", or if it’s just a result of random fluctuation in the data.

Hypothesis testing can be an integral component of any decision making process. It provides a framework for evaluating how confident one can be in making conclusions based on data. Some instances where this might come up include:
* a professor expects an exam average to be roughly 75%, and wants to know if the actual scores line up with this expectation. Was the test actually too easy or too hard?
* a product manager for a website wants to compare the time spent on different versions of a homepage. Does one version make users stay on the page significantly longer?

In this lesson, you will cover the fundamental concepts that will help you run and evaluate hypothesis tests:
* Sample and Population Mean
* P-Values
* Significance Level
* Type I and Type II Errors

You will then learn about three different hypothesis tests you can perform to answer the kinds of questions discussed above:
* One Sample T-Test
* Two Sample T-Test
* ANOVA (Analysis of Variance)

Let's get started!

Introduction

Suppose you want to know the average height of an oak tree in your local park. On Monday, you measure `10` trees and get an average height of `32` ft. On Tuesday, you measure `12` different trees and reach an average height of `35` ft. On Wednesday, you measure the remaining `11` trees in the park, whose average height is `31` ft. The average height for all `33` trees in your local park is `32.8` ft.

The collection of individual height measurements on Monday, Tuesday, and Wednesday are each called samples. A *sample* is a subset of the entire population (all the oak trees in the park). The mean of each sample is a *sample mean* and it is an estimate of the *population mean*.

Note: the sample means (`32` ft., `35` ft., and `31` ft.) were all close to the population mean (`32.8` ft.), but were all slightly different from the population mean and from each other.

For a population, the mean is a constant value no matter how many times it's recalculated. But with a set of samples, the mean will depend on exactly which samples are selected. From a sample mean, we can then extrapolate the mean of the population as a whole. There are three main reasons we might use sampling:

- data on the entire population is not available
- data on the entire population is available, but it is so large that it is unfeasible to analyze
- meaningful answers to questions can be found faster with sampling

Sample Mean and Population Mean - I

In the previous exercise, the sample means you calculated closely approximated the population mean. This won't always be the case!

Consider a tailor of school uniforms at a school for students aged `11` to `13`. The tailor needs to know the average height of all the students in order to know which sizes to make the uniforms.

The tailor measures the heights of a random sample of `20` students out of the `300` in the school. The average height of the sample is `57.5` inches. Using this sample mean, the tailor makes uniforms that fit students of this height, some smaller, and some larger.

After delivering the uniforms, the tailor starts to receive some feedback &mdash; many of the uniforms are too small! They go back to take measurements on the rest of the students, collecting the following data:
* 11 year olds average height: `56.7` inches
* 12 year olds average height: `59` inches
* 13 year olds average height: `62.8` inches
* All students average height (population mean): `59.5` inches

The original sample mean was off from the population mean by `2` inches! How did this happen?

The random sample of `20` students was skewed to one direction of the total population. More `11` year olds were chosen in the sample than is representative of the whole school, bringing down the average height of the sample. This is called a _sampling error_, and occurs when a sample is not representative of the population it comes from. How do you get an average sample height that looks more like the average population height, and reduce the chance of a sampling error?

Selecting only `20` students for the sample allowed for the chance that only younger, shorter students were included. This is a natural consequence of the fact that a sample has less data than the population to which it belongs. If the sample selection is poor, then you will have a sample mean seriously skewed from the population mean.

There is one surefire way to mitigate the risk of having a skewed sample mean — take a larger set of samples! The sample mean of a larger sample set will more closely approximate the population mean, and reduce the chance of a sampling error.

Sample Mean and Population Mean - II

You know that a hypothesis test is used to determine the validity of a null hypothesis. Once again, the null hypothesis states that there is no actual difference between the two populations of data. But what result does a hypothesis test actually return, and how can you interpret it?

A hypothesis test returns a few numeric measures, most of which are out of the scope of this introductory lesson. Here we will focus on one: p-values. P-values help determine how confident you can be in validating the null hypothesis. In this context, a _p-value_ is the probability that, assuming the null hypothesis is true, you would see at least such a difference in the sample means of your data.

Consider the experiment on history and chemistry majors and their interest in volleyball from a previous exercise:
* Null Hypothesis: `"History and chemistry students are interested in volleyball at the same rates"`
* Experiment Sample Means: `34%` of history majors and `39%` of chemistry majors sign up for the volleyball class 

Assuming the null hypothesis is true, there is no actual difference in preference for volleyball between all history and chemistry majors, and any difference present in the experiment data is the result of sampling error. Imagine you run a hypothesis test on this experiment data and it returns a p-value of `0.04`. A p-value of `0.04` indicates that you could expect to see a difference of at least `5%` (calculated as `39%` - `34%` = `5%`) in the sample means only 4% of the time.

Essentially, if you ran this same experiment `100` times, you would expect to see as large a difference in the sample means only `4` times given the assumption that there is no actual difference between the populations (i.e. they have the same mean).

Seems like a really small probability, right? Are you thinking about rejecting the null hypothesis you originally stated?

P-Values

Before you use numerical hypothesis tests, you need to be sure that the following things are true:

#### 1. The samples should each be normally distributed...ish

Data analysts in the real world often still perform hypothesis tests on datasets that aren't exactly normally distributed. What is more important is to recognize if there is some reason to believe that a normal distribution is especially unlikely. If your dataset is definitively not normal, the numerical hypothesis tests won't work as intended.

For example, imagine you have three datasets, each representing a day of traffic data in three different cities. Each dataset is independent, as traffic in one city should not impact traffic in another city. However, it is unlikely that each dataset is normally distributed. In fact, each dataset probably has two distinct peaks, one at the morning rush hour and one during the evening rush hour. The histogram of a day of traffic data might look something like this:

![histogram](https://content.codecademy.com/courses/learn-hypothesis-testing/lesson_ii/histogram_data_traffic.svg)

In this scenario, using a numerical hypothesis test would be inappropriate.

#### 2. The population standard deviations of the groups should be equal

For ANOVA and Two Sample T-Tests, using datasets with standard deviations that are significantly different from each other will often obscure the differences in group means.

To check for similarity between the standard deviations, it is normally sufficient to divide the two standard deviations and see if the ratio is "close enough" to 1. "Close enough" may differ in different contexts, but generally staying within `10%` should suffice.

#### 3. The samples must be independent

When comparing two or more datasets, the values in one distribution should not affect the values in another distribution. In other words, knowing more about one distribution should not give you any information about any other distribution.

Here are some examples where it would seem the samples are not independent:

* the number of goals scored per soccer player before, during, and after undergoing a rigorous training regimen
* a group of patients' blood pressure levels before, during, and after the administration of a drug

It is important to understand your datasets before you begin conducting hypothesis tests on them so that you know you are choosing the right test.

Assumptions of Numerical Hypothesis Tests

Suppose you want to know if students who study history are more interested in volleyball than students who study chemistry. Before doing anything else to answer your original question, you come up with a null hypothesis: `"History and chemistry students are interested in volleyball at the same rates."`

To test this hypothesis, you need to design an experiment and collect data. You invite `100` history majors and `100` chemistry majors from your university to join an extracurricular volleyball team. After one week, `34` history majors sign up (`34%`), and `39` chemistry majors sign up (`39%`). More chemistry majors than history majors signed up, but is this a “real”, or significant difference? Can you conclude that students who study chemistry are more interested in volleyball than students who study history?

In your experiment, the `100` history and `100` chemistry majors at your university are samples of their respective populations (all history and chemistry majors). The sample means are the percentages of history majors (`34%`) and chemistry majors (`39%`) that signed up for the team, and the difference in sample means is `39%` - `34%` = `5%`. The population means are the percentage of history and chemistry majors worldwide that would sign up for an extracurricular volleyball team if given the chance.

You want to know if the difference you observed in these sample means (`5%`) reflects a difference in the population means, or if the difference was caused by sampling error, and the samples of students you chose do not represent the greater populations of history and chemistry students.

Restating the null hypothesis in terms of the population means yields the following:

`"The percentage of all history majors who would sign up for volleyball is the same as the percentage of all chemistry majors who would sign up for volleyball, and the observed difference in sample means is due to sampling error."`

This is the same as saying, “If you gave the same volleyball invitation to every history and chemistry major in the world, they would sign up at the same rate, and the sample of `200` students you selected are not representative of their populations.”