Prusak.com

Serving up my thoughts since 2004

Conversion Confidence Level Calculator

One of the most common questions when setting up an experiment in Google Website Optimizer is how long will the experiment take?

In other words, how long will it take to have “enough” data.

The official Google Website Optimizer docs say:
https://www.google.com/analytics/siteopt/siteopt/help/techoverview.html

If one combination emerges as a clear winner, in the future it will exceed the performance of any of the other combinations with a 95% confidence or better.

But what do you do if you want to estimate the experiment length in advance?

Just use my conversion confidence level calculator

Here is a simple example:

Your site gets 2,000 visitors a week.
If you’re doing a simple A/B split with just two versions (original and test A) then each one gets 1,000 visitors a week.

Your current conversion rate is 2% * 1,000 visitors = 20 conversions a week for the original version.
You are hoping to get a 50% improvement for a 3% conversion rate * 1,000 visitors = 30 conversions a week for the test version.

Here’s what it will look like. The fields in yellow are the fields you enter.

The confidence calculator shows an 84% confidence level.
Not bad, but your boss says she wants at least a 95% confidence before declaring a winner.

You’ll see the confidence level for twice the traffic (two weeks worth of data) is 95%.
This means that if you get the same conversion rates and double the traffic, you’ll have a 95% confidence level.

– Ophir

1 Comment

  1. Hi Prusak

    As supplement to your explanation I can tell that the bars shown are the variables confidence interval. If only one part with two elements are tested then the bars shows the confidence interval for each of the elements. If the two bars don’t overlap then there is a significance difference on a 95% significance-level. Google write at which significance-level the results are significant, but the normal level is 95%.

    For further reading http://www.statpac.com/surveys/statistical-significance.htm

    Thanks for the programmer focus. As a number-nerd it is great to upgrade that 🙂

Leave a Reply to Jacob Kildebogaard Cancel reply

Your email address will not be published.

*

© 2024 Prusak.com

Theme by Anders NorenUp ↑