I don’t believe in A/B testing

Some people love to test everything from different designs to the color on buttons. They think it gives them special insight in how the vast anonymous Internet audience behaves; that they’ve cracked the code.

So according to your numbers (you might have a huge quantity for all I care) green buttons are more successful than orange buttons. In doing what exactly? As a call-to- action to try something for free? Well doh. In what context is the call-to-action button, is it on a dedicated “try my product” page or is it on the frontpage? From what is the user’s source; AdWords, banners, offline ads, generic search? What is the user’s intent, motivation – what does he/she want to achieve?

Well you see, you don’t know. That’s the problem with marketing, one can only speculate. Also, you might have “figured” out how a single factor works better than another single factor in an unique environment (a certain layout, certain color palette, certain font styles and so on) with several other unknown factors including user preferences, intentions, motivation and so on.

The point being; a test result is not certain to work in a different setting, and there’s quite a lot of ’em settings out there in the (world) wide web.

This entry was posted in Testing and tagged , , , , , , , , , , , , , , , , , , , . Bookmark the permalink.