Excellent work! In this lesson you have:
- Learned to interpret a p-value using a significance threshold
- Explored type I and type II errors in hypothesis testing
- Shown that the significance threshold for a hypothesis test is equal to the type I error rate
- Seen how multiple hypothesis tests lead to an increased probability of a type I error
In the workspace, we’ve borrowed a comic strip from xkcd that describes (with a little humor!) the problem of multiple hypothesis tests. The initial question asked in the comic is whether or not there is a link between jelly beans and acne. At first, scientists find no link — but then they run 20 more hypothesis tests looking for a link between specific colors of jelly beans and acne. One of those tests results in a significant p-value — and the final panel of the comic shows that this one test is the only test that gets published. The comic perfectly describes how running multiple hypothesis tests and only reporting significant results is extremely problematic. This has contributed to the reproducibility crisis in statistics. At a .05 significance level, approximately 5% of hypothesis tests will result in a false positive; but if we only report those false positives, we end up with a lot of published results that can’t be replicated.