In this lesson, we’ll use hypothesis tests to make inference about population-level associations between two variables.
We will cover four different hypothesis tests:
- Two Sample T-Tests (for an association between a quantitative variable and a binary categorical variable)
- ANOVA and Tukey Tests (for an association between a quantitative variable and a non-binary categorical variable)
- Chi-Square Tests (for an association between two categorical variables)
The applet provided for you in the workspace simulates the following scenario:
- A Project Manager (PM) wants to know whether website visitors spend more time on one version of a homepage (version 1) or another (version 2).
- In order to address this question, they randomly assign a sample of visitors to see one version or the other and record how long each visitor spends on the site.
- Then, the PM runs a two-sample t-test with a significance threshold of 0.05. A p-value less than 0.05 would suggest that visitors are spending significantly different amounts of time on the two homepages.
In the applet, you get to control the population level values! In other words, if the PM could show the two versions of the homepage to every possible past and future website visitor, you get to decide what the average amount of time spent on each version would be.
When you click “Generate”, the applet will re-sample a new set of visitors from your chosen populations (using the sample size that you specify), run a 2-sample t-test, and determine whether the means are significantly different.
Try the following:
- Press “Generate” a few times. Note that these sample distributions are different enough that the p-value is significant every time.
- Change the mean of version 2 so that it is equal to the mean for version 1. Press “Generate” a few times and note how the p-values are now mostly not significant.
- Change the means of version 1 and version 2 so that they are only different by 0.5 (eg., 2.5 and 3), but leave the standard deviations set to 1 for both groups. Press “Generate” a few times and notice that the test still detects a difference most of the time.
- Now change the standard deviations to 3 for both versions and press “Generate”. Notice how the sample distributions are now much flatter (there is more variation). Press “Generate” a few times and notice how the test no longer finds a difference every time, even though the difference is still 0.5.
- Play around with the sample size, standard deviations, and means. What kinds of differences can the test detect? Notice how both the magnitude of differences and variation in the data plays a role.