A/B Testing has quickly become one of the most important skills for marketers to master in order to improve performance of their marketing campaigns.

But, like anything else that grows quickly on the web, it’s easy to jump head first into optimization and A/B testing without first taking a step back to really think through how you’re going to get the most out of it. How do you treat your testing and optimization like a “project” and maximize your investment to get the most of your time?

In this article, I’ll give some tips on how to maximize your time investment when conducting A/B tests. This should help you stop “spinning your wheels” and actually be able to get measurable impact from

Simple Example

Let’s say that you are the marketing manager for the following site:

Abacus Example Site

Abacus 24/7 was featured on the Inc 500 listing 2009 revenue at about $8M. We’ll assume they’ve seen some growth since then and put them at $10M.

Looking at this site, they are clearly an excellent candidate for optimization. The sky is the limit here in things that you could test.

So, how do you decide what you’re going to go after first?

Low Hanging Fruit

Among CRO experts I’ve found two schools of thought:

  1. Test Simple Changes First: These are the “quick wins” that are usually obvious when an experience optimizer takes a look at your site. Typically you can expect a 10 – 20% improvement with quick wins.
  2. Test Dramatic Changes First: This lets you see the highest uplift in conversion rate, and then further refine these “drastic” changes with small tweaks to keep pushing conversion higher.

I find that most business owners are happier to see immediate improvements in conversion rate at a low cost rather than taking a “risk” and aiming for higher dramatic changes that may or may not improve conversion.

So, in this case, we’ll target the low hanging fruit first.

Prioritizing Tests

Before you start any project, it’s critical to prioritize tests and estimate the ROI of testing. If you skip this step, you’re running the risk that you’re going to “spin your wheels” and risk wasting a large time investment on tests that really aren’t going to drive the type of return that you need.

Using this website as an example, here’s an example:

  1. Shipping Offer – 10%: The site has an extremely strong offer for shipping – $5 Flat Rate Shipping and Free Shipping over $100. However, both of these offers are hidden on the side of the site and not made prominently enough. Placing shipping in the top of the header would likely boost conversion rate by at least 10% and would be the easiest “quick win” for this site to test.
  2. Reducing Choice – 10%: There are far too many products on this homepage. When someone comes on the site for the first time, they are required to choose between 6 different categories of items, each with another 5+ sub-items. Reducing the amount of products on the homepage would lift conversion by 10% or more.
  3. Removing Distractions and Text – 10%: Underneath the product images, there is a “ton” of text. This is in addition to the needless social icons on the right hand side along with the live chat and other multiple calls to action. Removing these would drive at least another 10% improvement.

Hopefully you can see now why quick wins are the easiest place to start. Multiplying each of these improvements together would lead to a 33% improvement in conversion rate. At $10M in revenue, that’s likely $3.3M in additional revenue from three simple tests. (This assumes that the combination of three variables will “combine” to lift conversion – more on that later)

Estimating Pre-Work

Once you’ve made the business case for testing, the next step is to think about how much implementation time you’ll need, as well as how much time you’re going to need to measure significance. This sounds simple but getting into the weeds can get complex quickly.

Here’s an example of what I mean:

In order to execute the tests above, there’s going to be some pre-work:

  1. Installation of Testing Code
  2. Review of Analytics Account (to determine average order value, etc)
  3. Internal Communication

Most of this sounds simple but likely is going to take 1-2 weeks depending on the size of the organization that you’re working with.

Estimating Time Required for Stat Significance & Traffic Fluctuation

Now that you’ve gotten the grunt work out of the way, I can’t emphasize enough how important it is for you to estimate the time required to reach statistical significance.

There’s a number of tools that you can use to guess how much time you’ll need to reach statistical significance – my favorite is this calculation tool from AB Tester – and it’s absolutely critical that you do these estimates in advance to help determine if testing is worth it and how much time testing is going to take.

For example –

The tests above each are simple, and for a site driving this much revenue, you wouldn’t need longer than 5 days to determine at a significant level whether the improvements that you have are working or not.

However, what happens when you want to do a multivariate test?

For the tests that we laid out above, there are several variations that you can run:

  1. Shipping Alone
  2. Shipping + Less Choice
  3. Shipping + Less Distractions

etc. etc. Is it worth it to go all out and do a full multivariate test on three variables, testing all combinations, or to simply assume that, because each variable lifted conversion individually, that the combination of all three will have the highest lift?

In my experience, the follow up multivariate tests are what most people forget to factor in as the time required to get true results.

Final Proposal

For this site then – even just going with the low hanging fruit – your final time investment is looking like this

  • Set Up – 1-2 Weeks
  • A/B Tests – 1-2 Weeks
  • Multivariate Tests – 1-2 Weeks
  • Implementation – 1-2 Weeks

That means that even the low hanging fruit is going to take 1-2 months of effort before you get into the “radical” testing changes you need to see the highest possible lift.


It’s easy to jump head first into A/B testing, but it’s critical that you invest the time required upfront to layout your testing ideas and estimate the true total time investment. This lets you manage expectations with your team and also quickly determine whether the labor invested in a project is worth the effort.

How have you approached A/B testing in the past?

Featured images:
  •  License: Image author owned

About The Author

Andy Hunt worked at Google for 3 years before starting Uplift ROI. His specialties among others are conversion rate optimization, search engine marketing and social media marketing. Follow him on Twitter @UpliftROI to get updated on all the latest in CRO and internet marketing.