An A/B Test is a scientific method of choosing between two options (Option A and Option B). Some examples of A/B tests include:
- What number of sale items on a website makes customers most likely to purchase something: 25 or 50?
- What color button are customers more likely to click on: blue or green?
- Do people spend more time on a website if the background is green or orange?
For A/B tests where the outcome of interest (eg., whether or not a customer makes a purchase) is categorical, an A/B test is conducted using a Chi-Square hypothesis test. In order to determine the sample size necessary for this kind of test, a sample size calculator requires three numbers:
- Baseline conversion rate
- Minimum detectable effect (also called the minimum desired lift)
- Statistical significance threshold
In this lesson, we will discuss each of these numbers and how a data scientist might choose them.
Take a look at the sample size calculator provided in the workspace and try plugging in some numbers. Can you guess what each input means?