One of the limitations of using a standard A/B split test is that it won’t work out of the box if the page you are testing is the “action” page for a form that uses method=POST.
For example, lets say you have a simple form page – form.html that looks something like this:
You’ve created process_form_2.php to split test against process_form_1.php.
The problem is that when process_form_1.php redirects to process_form_2.php, you will have lost all of your form data since your form uses POST.
There are two solutions for this:
1. Switch from method=POST to method=GET.
If this is an option, it’s probably the easiest solution.
2. Setup the experiment as a multivariate experiment and change the form action line in the HTML.
Someone recently asked on the GWO form:
Why would 10% expected improvement take a lot longer than a 20%?
I would think it is the other way around.
The truth is – the smaller the expected improvement, the longer it takes.
Or, in other words, the bigger the difference in conversion between two variations, the shorter time it will take to see the difference.
Here is an analogy that will hopefully provide an intuitive asnwer.
Lets say I want to test two basketball teams against each other, seeing who’s the better team.
Team A currently wins 20% of the time.
I want to test if team B is a 100% improvement over team A, which would mean team B wins 40% of the time.
The question is, how many games do they need to play to be 95% sure than group B is indeed better. (95% is the confidence level GWO uses).
Without going into the math, if team B indeed is twice as good as team A, I should know fairly quickly.
Big differences are obvious in a short time.
On the other hand …
I want to test if team B is just 10% better than team A, which would mean team B wins 22% of the time.
How many games do they need to play to be 95% sure than group B is indeed better.
Again, without going into the math, if team B indeed is just 10% better than team A, they’ll need to play quite a lot of games to be 95% certain that they are indeed 10% better.
Small differences take longer to notice than big differences.