Suppose we’re running an A/B Test to find out if a new website layout drives more subscriptions than the current one. If the new layout is only a tiny percent better, would we really care?
In order to detect precise differences, we need a very large sample size. In order to choose a sample size, we need to know the smallest difference that we actually care to measure. This “smallest difference” is our desired minimum detectable effect. This is also sometimes referred to as desired lift.
Minimum detectable effect or lift is generally expressed as a percent of the baseline conversion rate. Suppose that 6% of customers currently subscribe to our website (that’s our baseline conversion rate). Changing a website layout is hard, so we only think that it’s worth doing if at least 8% of our customers would subscribe with the new layout. To calculate this as a percentage of our baseline:
baseline = 6 new = 8 min_detectable_effect = (new - baseline) / baseline * 100 print(min_detectable_effect) #output: 33.0
Our minimum detectable effect/desired lift is 33%.
Suppose you have a baseline conversion rate of
8% and want to implement a new website design if it will increase the conversion rate to
12%. What is your minimum detectable effect? Save that number into a variable called
mde and print it out.