Congratulations! You now know how to use a sample size calculator to figure out an appropriate sample size for an A/B Test! As a reminder, you learned about the following inputs to a sample-size calculator:

  • Baseline Conversion Rate
  • Minimum Detectable Effect
  • Significance Threshold


As a final exercise, let’s put everything together into a single calculation. Suppose that you are running a business and want to see if a new advertisement will drive more clicks on your website. Currently, about 10% of people who see your ad are clicking on it. You want to run the new ad if at least 14% of people will click the new ad. When you run your Chi-Square test after collecting your data, you plan to use a significance threshold of 0.05, so that your chances of a false positive are relatively low. Try the following:

  • Based on the description above, identify the baseline conversion rate and significance threshold
  • Based on the description above, calculate the minimum detectable effect (hint: it’s not 4%!)
  • Plug in your baseline, minimum detectable effect, and significance threshold to the provided calculator
  • Calculate the total sample size needed for this experiment (note: this calculator assumes that exactly half of the sample will see each version of the ad)

The solution to this problem is located in solution.py if you’d like to check your work.

Take this course for free

Mini Info Outline Icon
By signing up for Codecademy, you agree to Codecademy's Terms of Service & Privacy Policy.

Or sign up using:

Already have an account?