There’s a new book in town.
Always Be Testing:
The Complete Guide to Google Website Optimizer
While the book was being written a few months ago, I had the opportunity to discus the technical side of using Google Website Optimizer with one of the authors, John Quarto-vonTivadar.
The book mentions me in a few places and lists www.prusak.com in the resources section 🙂
I’m currently working on something and I can’t go into details but I’ll have an update soon.
A few months ago a created a Google Website Optimizer Video Tutorial
I really didn’t do much with it, and since then they’ve slightly updated the interface.
The good thing is that it provides a step-by-step guide explaining exactly how to setup a test, as well as some interesting background information and common mistakes to avoid.
If you find it helpful, please let me know.
Check it out Here.
One of the most common questions when setting up an experiment in Google Website Optimizer is how long will the experiment take?
In other words, how long will it take to have “enough” data.
The official Google Website Optimizer docs say:
If one combination emerges as a clear winner, in the future it will exceed the performance of any of the other combinations with a 95% confidence or better.
But what do you do if you want to estimate the experiment length in advance?
Just use my conversion confidence level calculator
Here is a simple example:
Your site gets 2,000 visitors a week.
If you’re doing a simple A/B split with just two versions (original and test A) then each one gets 1,000 visitors a week.
Your current conversion rate is 2% * 1,000 visitors = 20 conversions a week for the original version.
You are hoping to get a 50% improvement for a 3% conversion rate * 1,000 visitors = 30 conversions a week for the test version.
Here’s what it will look like. The fields in yellow are the fields you enter.
The confidence calculator shows an 84% confidence level.
Not bad, but your boss says she wants at least a 95% confidence before declaring a winner.
You’ll see the confidence level for twice the traffic (two weeks worth of data) is 95%.
This means that if you get the same conversion rates and double the traffic, you’ll have a 95% confidence level.
I came across this question on the Google Website Optimizer group:
Hi, I run a website where I sell 7 different products each at a different price level.
I am wondering whether there’s a way so that I can track the changes I make in terms of their effect on my overall revenue?
From a marketing perspective, this raises an interesting question.
Will the test version with the best conversion rate always bring the most revenue?
If what you are testing has no relationship to what you are selling, then probably yes.
On the other hand, lets say you want to test what product you should be highlighting.
In such a case, measuring overall revenue makes sense.
So here is how I would do it…
One of the limitations of Google Website Optimizer is that you have no control over which combination will be displayed to a user.
Wouldn’t it be great to have total control over which variation is displayed in each section?
For example, lets say you want to test a new navigation scheme that required testing of the page header and footer, but they had to be in sync with each other. In other words, header #1 must be displayed with footer#1, header #2 must be displayed with footer#2 and so forth.
This “hack” gives you the ability to read and write which variation is displayed in each section.
I just received an email with this question:
I’m trying to set up an GWO A/B test, but I’m having a problem
completing the set-up of the test because the confirmation page
requires a login. It’s not a public page because there’s sensitive
order info on there. So we’ve got the CWO code posted on the
confirmation page but GWO can’t validate it because it’s not
a public page.
fire and it should work okay. Any idea how I can complete the set-up
of the test without going through successful validation? Right now I’m
stuck because I can’t hit the continue button in GWO to get past the
“Validate pages” step.
So I checked and indeed Jonathan is right.
You can’t continue without a “live” conversion page that is publicly accessible.
When creating a Multivariate experiment, this is a non-issue because you can upload the conversion page from your local file-system, bypassing the need for the file to be publicly accessible.
Hope that helps
I’ve written in past posts about how to get all of the great GA data for each of your combinations in GWO.
A couple of months ago I ran an experiment that showed me why doing so isn’t just a “nice thing to have”, but a necessity.
I was comparing two significantly different versions of a promotion page (whole page split). You can see them here:
Old (control) Version
After a couple of weeks of testing, GWO was reporting that the new version was performing 7.03% worse than the old version. I was a bit surprised because of the radical difference between the two pages.
At that point I started looking at numbers GA provided as well. At first I didn’t see anything out of the ordinary. Then I took a look at the page performance segmented by visitor type (New / Return visitors) and my eyes opened wide.
For new visitors, the new version of the page was performing much better (over 40% improved conversion rate) and for return visitors the old version of the page was performing much better (over 20% better conversion rate).
Going forward, we now show new users the new version of the page and return users the old version. I have a few theories on why we’re seeing this behavior, but that’s a bit beyond the scope of this post. I’d love to hear your ideas in the comments.
What this shows us is that while GWO is an excellent tool, it can be a bit blunt when used on it’s own. Very often the most valuable information is in the details.
UPDATE: (Sep 6, 2007) Google just redid the entire A/B split testing process. See:
While GWO’s was originally created to test specific elements within a page, sometimes you want to test totally different versions of a page.
Lets say you want to test three totally different versions of page1.html. You create three new pages: page1a.html, page1b.html and page1c.html. You and then add some custom code to page1.html which redirects visitors to one of the new pages.
GWO added this feature a while back. You can read the official page at Comparing entire pages (A/B testing)
Recently, one of the posts on the Official GWO Forum mentioned a shortcoming of Google’s solution – the fact that it doesn’t carry forward the query string.
Lets say the URL of your main test page is
landing.html and one of your test pages is
If a user is sent to
landing.html?code=123 they will be forwarded to
landing_1.html and not
landing_1.html?code=123 which is what you probably wanted.
This is easier then you might think.
Look for this line in Google’s original code for your test page:
document.location = test_url;
and change it to:
document.location = test_url + self.location.search;
A few weeks ago Shawn Purtell from ROI Revolution wrote an excellent article on combining GWO with GA. If you haven’t read his article – please do so now. This posting describes a bug fix for his code, but you really should first read his article to fully understand how to use it.
I just found a bug in his code and am providing a fix for it. Under some circumstances, his code will provide the wrong combination number for the combination that is being displayed. Different combinations can be given the same combination number.
Google just announced a new and improved version of their analytics software: