Skip to Content
Learn
A/B Testing: Sample Size Calculators
Introduction

An A/B Test is a scientific method of choosing between two options (Option A and Option B). Some examples of A/B tests include:

  • What number of sale items on a website makes customers most likely to purchase something: 25 or 50?
  • What color button are customers more likely to click on: blue or green?
  • Do people spend more time on a website if the background is green or orange?

For A/B tests where the outcome of interest (eg., whether or not a customer makes a purchase) is categorical, an A/B test is conducted using a Chi-Square hypothesis test. In order to determine the sample size necessary for this kind of test, a sample size calculator requires three numbers:

  • Baseline conversion rate
  • Minimum detectable effect (also called the minimum desired lift)
  • Statistical significance threshold

In this lesson, we will discuss each of these numbers and how a data scientist might choose them.

Instructions

Take a look at the sample size calculator provided in the workspace and try plugging in some numbers. Can you guess what each input means?

Folder Icon

Sign up to start coding

Already have an account?