The Science Behind A/B Testing

By / Sep 9, 2011 / Tips
shares

If you own a business, there’s always concern that your marketing efforts aren’t reaching their full potential. Of course, great design will sell but having strategies and formulas in place to make sure your hard work is paying off it well worth the time. One of these formulas is A/B testing. A/B Testing, or split testing, has been enacted in the past few years by companies who want their designs to go under experimentation as an approach to their marketing decisions. It’s a simple practice that companies that to maximize their profits. In this article I will uncover the science behind this approach to design decision making, giving you the tools necessary to make decisions on whether you want to incorporate this into your own design process.

A/B Testing performed on a website

What is A/B Testing?

A/B Testing is a method of marketing testing by which two versions of one element are tested against a metric to define which is more successful. These versions undergo testing simultaneously to determine which is better. Conversions are measured from two different sets of users. The version more effective is then used for the companies marketing initiatives. The result is high-performance user experience elements that aren’t always achieved by just relying on instinct.

A/B tests can be used on a myriad of elements in a design. For example if you wish to test the effectiveness of a call-to-action button, have it undergo an A/B against a new version to see which is more successful. This is especially effective if your goal is to increase the number of conversions as you’ll get an idea of what prevents that person from converting.

You can also compare the following using A/B tests:

    • Headlines
    • Call to action text, wording, size, and placement
    • Length of body copy
    • Layout and style of a website
    • Images on landing and product pages
    • etc.

    A/B Testing can create a lasting impact on your marketing efforts. Knowing which elements work the best is critical in moving forward as you can repeat successful tests in the future. It also gives decision-makers a very good picture of how their designs are doing. If simply conducting an A/B test had the ability to increase profits and revenue of a business, it’ll be hard for a businessperson to say no. Not only are you measuring design but the conversions you measure can be clicks on a link, rate of returns on a site, and product purchases. In the end it’s a win-win situation for both the designer and client as your designs have the data to back them up rather than relying solely on opinion.

    testing a landing page

    What to Test

    In determining what to test it’s important to have the versions be different enough that you’ll be able to gain insight into the correct procedures going forward. We’re not talking about slight differences between the two, such as a text color difference or font weight variation.

    It’s also important to only test one variable at a time. If you throw in multiple variables, it’ll be hard to access just what works correctly and what doesn’t.

    A good A/B Test could compare:

    • Requiring a sign-up for some visitors but optional for others
    • Completely different email subject lines

    Running It

    The process continues by running the test on your site. Explore different testing tools according to your platform or medium for the right solution. Ideally, you’ll want to be able to control:

    • the number of visitors who see each version of the test,
    • the difference between each version, and
    • how you measure the effect of each test.

    After a few weeks you’ll want to check the results of each version. For the most exposure, try and run the test on the high-traffic areas of your site. Before you can trust the result of your A/B test, it’s important to you have the right statistics. A calculator such as Usereffect.com’s will give you a good idea of where you are and where you should be.

    Advantages of A/B Testing

    Some advantages of A/B Testing include the following:

    • Low risk approach
    • Cheaper than other methods such as focus groups
    • Provides proof
    • Invisible to most of your users
    • Great way to do ‘test run’ new designs before full roll-out (to avoid negative surprises on the launch day
    • Can solve internal disputes

    Increase Conversion Icon

    Conclusion

    A/B Testing is a powerful, formulaic approach to testing out two different designs. In an era of usability and accessibility, it’s becoming a more and more popular method to ensure your design is converting. It’s important though to give your tests some time – ideally a few weeks before you check the data. If you’ve come to a fork in the road in terms of a design direction, A/B tests are a great solution to see just what version is more successful. Do you conduct A/B tests and are they useful for you? Leave comments below telling us how.

    About the Author

    Stephanie Hamilton is a freelance Graphic and Web Designer by day and a blogger by night. Holding a BFA in Graphic Design, she has several years experience designing the latest in print and building better user interfaces and websites. She also enjoy writing as a creative outlet and blog regularly for her business and contribute to others as well. You can visit her website and follow her on twitter.

9 Comments

  1. Kyle
    September 9, 2011

    Don’t forget to proof read. ;)

    Reply
  2. Rancid
    September 10, 2011

    Great tips regarding testing designs. I personally never have the patience to wait a few days when compering 2 different designs in one of my websites. That’s why I never get accurate data and I always end up using the design I love the most.

    Reply
  3. isadora
    September 11, 2011

    great tips – thanks for the post

    Reply
  4. Simon Day
    September 13, 2011

    A/B testing is critical if you want to maximise sales. Very few clients are prepared to pay for it and yet I’ve seen 200%+ increase in sales by fine-tuning the page. Even something as simple as changing the colour of the call-to-action button can really bosst sales.

    You didn’t offer any links to services which can help in this. May I suggest Google’s own A/B testing tool at: http://www.google.com/websiteoptimizer/ as a starter.

    Reply
    • Tim
      October 4, 2011

      Another great testing tool that I use is Zentester, it’s free and I switched to it from Google Website Optimizer because it’s so much easier to use (visual testing).

      Reply
  5. Hameed Rahamathullah
    September 14, 2011

    Very crispy post on A/B testing.

    Reply
  6. Vitaly
    September 27, 2011

    http://headmetrics.com/ – cheap and easy a/b testing tool

    Reply
  7. Web Design Washington
    November 27, 2011

    These are great tips about A/B testing. Thanks for the share.

    Reply
  8. SEO Surrey
    December 28, 2011

    A / B split testing. What a fantastic article on testing. This makes me think back of the heady days of working in a marketing department. Stop, I feel a little dizzy! Finished? Finished you say? Go do some more testing!

    Reply

Leave a Reply