Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-settings.php on line 512

Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-settings.php on line 527

Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-settings.php on line 534

Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-settings.php on line 570

Strict Standards: Declaration of Walker_Page::start_lvl() should be compatible with Walker::start_lvl(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1199

Strict Standards: Declaration of Walker_Page::end_lvl() should be compatible with Walker::end_lvl(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1199

Strict Standards: Declaration of Walker_Page::start_el() should be compatible with Walker::start_el(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1199

Strict Standards: Declaration of Walker_Page::end_el() should be compatible with Walker::end_el(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1199

Strict Standards: Declaration of Walker_PageDropdown::start_el() should be compatible with Walker::start_el(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1244

Strict Standards: Declaration of Walker_Category::start_lvl() should be compatible with Walker::start_lvl(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1391

Strict Standards: Declaration of Walker_Category::end_lvl() should be compatible with Walker::end_lvl(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1391

Strict Standards: Declaration of Walker_Category::start_el() should be compatible with Walker::start_el(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1391

Strict Standards: Declaration of Walker_Category::end_el() should be compatible with Walker::end_el(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1391

Strict Standards: Declaration of Walker_CategoryDropdown::start_el() should be compatible with Walker::start_el(&$output) in /home/ophir/prusak.com/wp-includes/classes.php on line 1442

Strict Standards: Redefining already defined constructor for class wpdb in /home/ophir/prusak.com/wp-includes/wp-db.php on line 308

Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-includes/cache.php on line 103

Strict Standards: Redefining already defined constructor for class WP_Object_Cache in /home/ophir/prusak.com/wp-includes/cache.php on line 431

Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-includes/query.php on line 61

Deprecated: Assigning the return value of new by reference is deprecated in /home/ophir/prusak.com/wp-includes/theme.php on line 1109

Strict Standards: Declaration of Walker_Comment::start_lvl() should be compatible with Walker::start_lvl(&$output) in /home/ophir/prusak.com/wp-includes/comment-template.php on line 1266

Strict Standards: Declaration of Walker_Comment::end_lvl() should be compatible with Walker::end_lvl(&$output) in /home/ophir/prusak.com/wp-includes/comment-template.php on line 1266

Strict Standards: Declaration of Walker_Comment::start_el() should be compatible with Walker::start_el(&$output) in /home/ophir/prusak.com/wp-includes/comment-template.php on line 1266

Strict Standards: Declaration of Walker_Comment::end_el() should be compatible with Walker::end_el(&$output) in /home/ophir/prusak.com/wp-includes/comment-template.php on line 1266

Strict Standards: Redefining already defined constructor for class WP_Dependencies in /home/ophir/prusak.com/wp-includes/class.wp-dependencies.php on line 31

Strict Standards: Redefining already defined constructor for class WP_Http in /home/ophir/prusak.com/wp-includes/http.php on line 61

Warning: session_start(): Cannot send session cookie - headers already sent by (output started at /home/ophir/prusak.com/wp-settings.php:512) in /home/ophir/prusak.com/wp-content/plugins/wordpress-automatic-upgrade/wordpress-automatic-upgrade.php on line 92

Warning: session_start(): Cannot send session cache limiter - headers already sent (output started at /home/ophir/prusak.com/wp-settings.php:512) in /home/ophir/prusak.com/wp-content/plugins/wordpress-automatic-upgrade/wordpress-automatic-upgrade.php on line 92

Strict Standards: Redefining already defined constructor for class ftp_base in /home/ophir/prusak.com/wp-content/plugins/wordpress-automatic-upgrade/lib/ftp_class.php on line 56

Strict Standards: Redefining already defined constructor for class ftp in /home/ophir/prusak.com/wp-content/plugins/wordpress-automatic-upgrade/lib/ftp_class_sockets.php on line 8
The #1 Split Testing Mistake

Prusak.com

Online Since 1996

The #1 Split Testing Mistake

Posted by ophir on December 18th, 2009

I was reading a simple case study today.
They were testing two different versions of a banner that was advertising a webinar.
One of the banners had an image of the presenter, while the other did not.
The banner without the image of the presenter won (by over 50%).

One of the comments was something along the lines of:
I guess this audience prefers banners without an image of a person.

*sigh*

If you don’t immediately realize the mistake the commenter made, don’t feel bad. It’s a very common mistake.

Beyond the fact that a specific banner (which did have have an image of the presenter) won over a different specific banner (which did not have an image of the presenter) you really can’t be sure of anything.

The loosing banner might have won with:

  • An image of a different person
  • A different image of the same person
  • The same image of the same person in a different position or size on the banner.
  • The same image of the same person in the same position and size but with different elements on the banner changed.

The point is:

Don’t jump to generalized conclusions based on the outcome of a specific experiment.

-Ophir

One Response to “The #1 Split Testing Mistake”

  1. dan Says:

    Excellent simple point about the fundemental nature of testing…we all need to come back to basic questions over and over.

    thanks

    Dan Shannon
    web
    marketing for film

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>