Suppose that a company is considering a new color-scheme for their website. They think that visitors will spend more time on the site if it is brightly colored. To test this theory, the company shows the old and new versions of the website to 50 site visitors, each — and finds that, on average, visitors spent 2 minutes longer on the new version compared to the old. Will this be true of future visitors as well? Or could this have happened by random chance among the 100 people in this sample?
One way of testing this is with a 2-sample t-test. The null hypothesis for this test is that average length of a visit does not differ based on the color of the website. In other words, if we could observe all site visitors in two alternate universes (one where they see each version of the site), the average visiting times in these universes would be equal.
We can use SciPy’s
ttest_ind() function to perform a 2-sample t-test. It takes the values for each group as inputs and returns the t-statistic (not covered in this course) and a p-value:
from scipy.stats import ttest_ind tstat, pval = ttest_ind(times_version1, times_version2)
ttest_ind() runs a two-sided test.
The company randomly sampled 100 site visitors. They showed the old version of their website to half of their sample and the new version to the other half. The amount of time (in minutes) that each visitor spent on the website was recorded.
An overlaid histogram showing the amount of time spent on the website by visitors to the new and old versions has already been created for you in script.py. Press “Run” and inspect the histograms. Based on this picture, do you think there is a significant association between the version of the website a visitor saw and how long they spent on the site?
The data from this study has already been saved for you in script.py: the time spent by the 50 visitors to the old version is saved as
old; the time spent by visitors to the new version is saved as
new. Run a two-sample t-test comparing these groups and save the p-value as
pval, then print it out.
Using a significance threshold of 0.05, is there a significant difference between the average amount of time visitors are spending on the old and new versions of the website? In script.py set the value of
significant equal to
True if there is a significant difference and
False if not.