Prusak.com

One of the most common questions when setting up an experiment in Google Website Optimizer is how long will the experiment take?

In other words, how long will it take to have “enough” data.

The official Google Website Optimizer docs say:
https://www.google.com/analytics/siteopt/siteopt/help/techoverview.html

If one combination emerges as a clear winner, in the future it will exceed the performance of any of the other combinations with a 95% confidence or better.

But what do you do if you want to estimate the experiment length in advance?

Just use my conversion confidence level calculator

Here is a simple example:

Your site gets 2,000 visitors a week.
If you’re doing a simple A/B split with just two versions (original and test A) then each one gets 1,000 visitors a week.

Your current conversion rate is 2% * 1,000 visitors = 20 conversions a week for the original version.
You are hoping to get a 50% improvement for a 3% conversion rate * 1,000 visitors = 30 conversions a week for the test version.

Here’s what it will look like. The fields in yellow are the fields you enter.

The confidence calculator shows an 84% confidence level.
Not bad, but your boss says she wants at least a 95% confidence before declaring a winner.

You’ll see the confidence level for twice the traffic (two weeks worth of data) is 95%.
This means that if you get the same conversion rates and double the traffic, you’ll have a 95% confidence level.

– Ophir