An A/B Test is a scientific method of choosing between two options (Option A and Option B). Some examples of A/B tests include:

  • What number of sale items on a website makes customers most likely to purchase something: 25 or 50?
  • What color button are customers more likely to click on: blue or green?
  • Do people spend more time on a website if the background is green or orange?

For A/B tests where the outcome of interest (eg., whether or not a customer makes a purchase) is categorical, an A/B test is conducted using a Chi-Square hypothesis test. In order to determine the sample size necessary for this kind of test, a sample size calculator requires three numbers:

  • Baseline conversion rate
  • Minimum detectable effect (also called the minimum desired lift)
  • Statistical significance threshold

In this lesson, we will discuss each of these numbers and how a data scientist might choose them.


Take a look at the sample size calculator provided in the workspace and try plugging in some numbers. Can you guess what each input means?

Take this course for free

Mini Info Outline Icon
By signing up for Codecademy, you agree to Codecademy's Terms of Service & Privacy Policy.

Or sign up using:

Already have an account?