A/B Testing has quickly become one of the most important skills for marketers to master in order to improve performance of their marketing campaigns.
But, like anything else that grows quickly on the web, it’s easy to jump head first into optimization and A/B testing without first taking a step back to really think through how you’re going to get the most out of it. How do you treat your testing and optimization like a “project” and maximize your investment to get the most of your time?
In this article, I’ll give some tips on how to maximize your time investment when conducting A/B tests. This should help you stop “spinning your wheels” and actually be able to get measurable impact from
Let’s say that you are the marketing manager for the following site:
Abacus 24/7 was featured on the Inc 500 listing 2009 revenue at about $8M. We’ll assume they’ve seen some growth since then and put them at $10M.
Looking at this site, they are clearly an excellent candidate for optimization. The sky is the limit here in things that you could test.
So, how do you decide what you’re going to go after first?
Low Hanging Fruit
Among CRO experts I’ve found two schools of thought:
- Test Simple Changes First: These are the “quick wins” that are usually obvious when an experience optimizer takes a look at your site. Typically you can expect a 10 – 20% improvement with quick wins.
- Test Dramatic Changes First: This lets you see the highest uplift in conversion rate, and then further refine these “drastic” changes with small tweaks to keep pushing conversion higher.
I find that most business owners are happier to see immediate improvements in conversion rate at a low cost rather than taking a “risk” and aiming for higher dramatic changes that may or may not improve conversion.
So, in this case, we’ll target the low hanging fruit first.
Before you start any project, it’s critical to prioritize tests and estimate the ROI of testing. If you skip this step, you’re running the risk that you’re going to “spin your wheels” and risk wasting a large time investment on tests that really aren’t going to drive the type of return that you need.
Using this website as an example, here’s an example:
- Shipping Offer - 10%: The site has an extremely strong offer for shipping – $5 Flat Rate Shipping and Free Shipping over $100. However, both of these offers are hidden on the side of the site and not made prominently enough. Placing shipping in the top of the header would likely boost conversion rate by at least 10% and would be the easiest “quick win” for this site to test.
- Reducing Choice – 10%: There are far too many products on this homepage. When someone comes on the site for the first time, they are required to choose between 6 different categories of items, each with another 5+ sub-items. Reducing the amount of products on the homepage would lift conversion by 10% or more.
- Removing Distractions and Text – 10%: Underneath the product images, there is a “ton” of text. This is in addition to the needless social icons on the right hand side along with the live chat and other multiple calls to action. Removing these would drive at least another 10% improvement.
Hopefully you can see now why quick wins are the easiest place to start. Multiplying each of these improvements together would lead to a 33% improvement in conversion rate. At $10M in revenue, that’s likely $3.3M in additional revenue from three simple tests. (This assumes that the combination of three variables will “combine” to lift conversion – more on that later)
Once you’ve made the business case for testing, the next step is to think about how much implementation time you’ll need, as well as how much time you’re going to need to measure significance. This sounds simple but getting into the weeds can get complex quickly.
Here’s an example of what I mean:
In order to execute the tests above, there’s going to be some pre-work:
- Installation of Testing Code
- Review of Analytics Account (to determine average order value, etc)
- Internal Communication
Most of this sounds simple but likely is going to take 1-2 weeks depending on the size of the organization that you’re working with.
Estimating Time Required for Stat Significance & Traffic Fluctuation
Now that you’ve gotten the grunt work out of the way, I can’t emphasize enough how important it is for you to estimate the time required to reach statistical significance.
There’s a number of tools that you can use to guess how much time you’ll need to reach statistical significance – my favorite is this calculation tool from AB Tester – and it’s absolutely critical that you do these estimates in advance to help determine if testing is worth it and how much time testing is going to take.
For example -
The tests above each are simple, and for a site driving this much revenue, you wouldn’t need longer than 5 days to determine at a significant level whether the improvements that you have are working or not.
However, what happens when you want to do a multivariate test?
For the tests that we laid out above, there are several variations that you can run:
- Shipping Alone
- Shipping + Less Choice
- Shipping + Less Distractions
etc. etc. Is it worth it to go all out and do a full multivariate test on three variables, testing all combinations, or to simply assume that, because each variable lifted conversion individually, that the combination of all three will have the highest lift?
In my experience, the follow up multivariate tests are what most people forget to factor in as the time required to get true results.
For this site then – even just going with the low hanging fruit – your final time investment is looking like this
- Set Up – 1-2 Weeks
- A/B Tests – 1-2 Weeks
- Multivariate Tests – 1-2 Weeks
- Implementation – 1-2 Weeks
That means that even the low hanging fruit is going to take 1-2 months of effort before you get into the “radical” testing changes you need to see the highest possible lift.
It’s easy to jump head first into A/B testing, but it’s critical that you invest the time required upfront to layout your testing ideas and estimate the true total time investment. This lets you manage expectations with your team and also quickly determine whether the labor invested in a project is worth the effort.
How have you approached A/B testing in the past?
- License: Image author owned
About The Author
Andy Hunt worked at Google for 3 years before starting Uplift ROI. His specialties among others are conversion rate optimization, search engine marketing and social media marketing. Follow him on Twitter @UpliftROI to get updated on all the latest in CRO and internet marketing.
The ever-present debate between drivers, pedestrians and cyclists regarding road safety and the potential for accidents has got me thinking; who is the biggest contributor to road accidents in the UK? A few years back the Daily Mail reported that around 1 in 20 road accidents that took place were related to speeding, which contradicts with the sudden explosion we’ve seen with regards to speed cameras popping up left, right and centre throughout the country.
What’s more, there have been conflicting opinions raised in the past concerning various aspects of speed cameras, some of which go as far as condemning their existence. The same report from the Daily Mail pointed out that a much larger portion of road casualties were actually the result of a lack of concentration from road users, including pedestrians.
So where does the debate currently stand with regards to speed cameras? They certainly represent the demands of an economical, well-regimented and practiced society but do they really have a significant effect on our overall safety once we get behind the wheel?
The Daily Mail report resulted in critics taking the opportunity to vent their frustration at the lack of credibility between road accidents and the government slogan “speeding kills”. Motorists lost well over £100 million as a result of speeding tickets in the same year, which raised concerns as to whether further investment in speed cameras would actually benefit our safety.
There was further confusion after the Department for Transport claimed 15 per cent, and not 5 per cent, of accidents were related to speeding. However, this figure also included drivers that were deemed to be travelling at speeds not fit for road conditions, despite crucially not breaking the speed limit.
Do Speed Cameras Help?
The Safe Speed campaign group have come forward to stress their concerns with the growing reliance on speed cameras. They believe that far too much attention is currently being focused on speeding and not necessarily on the safety of our roads.
In 2013, research was carried out by University College London on behalf of the RAC Foundation to see just how much of an impact speed cameras were having on the overall behaviour of drivers and the rate of road accidents across the country. In what turned out to be an intriguing display of opinions, newspapers were split as to whether the results indicated a positive or negative outcome.
The Telegraph was quick to point out in their report that speed cameras significantly reduced the number of road casualties and serious injuries by as much as 27 per cent. This was an encouraging headline that supported further investment in speed cameras, but the Daily Mail were more interested in another finding from the research.
21 of the 551 speed camera sites nationwide had fallen under scrutiny after the number of road accidents recorded in these specific areas increased. It was enough for the Daily Mail to brandish the headline “Speed Cameras Increase the Risk of Serious or Fatal Crashes” and it also resulted in the RAC Foundation claiming the results provided “enough to make the cameras worthy of investigation”.
Should Speed Cameras Be Scrapped?
Paul Smith, founder of the Safe Speed campaign, was quoted in the Daily Mail as saying “the concentration on speeding is a deadly mistake. Speed cameras should be scrapped”. These were strong words from the founder of a group that provides considerable awareness for road safety protocols.
Despite this, the director of the RAC Foundation, Stephen Glaister, felt that enough evidence had been put forward to support the effects of speed cameras. “Without speed cameras, there would be around 800 more people killed or seriously injured” he said.
Whilst the findings prove inconclusive when it comes to speed cameras actually causing accidents, you’d think that with the RAC Foundations research in hand we could probably rest assured that they are contributing in some way to the safety of our roads. Unfortunately, this contradicts another report that emerged back in 2010 regarding speed cameras and their effects in Swindon.
The report stated that fewer accidents were recorded when speed cameras were switched off in Swindon, compared to when they were active. Over a period of nine months when the cameras were switched off, there were 315 road casualties in Swindon compared to the 327 recorded during a similar time frame the year before.
As you can see, the research that’s been carried out in the past has brought about varied results that have consequently structured differing opinions, so would an increase in speed cameras and a further crackdown on speeding bring about the results drivers and pedestrians care about the most?
You’d have to say it would seem illogical. Speed cameras may save 800 lives every year but speeding only contributes to 5 per cent of road casualties. Despite the RAC Foundations research, it would seem that concentrating on other road safety protocols might have more of a significant effect on the wider scheme of things, such as eradicating mobile phone usage behind the wheel and multiplying the number of pedestrian crossings we currently have access to.
Many of us are reliant on our vehicles for business purposes and no matter how careful a driver we may be, extenuating circumstances may sometimes happen that cause us to fall foul of the law or experience a lapse in driver due care and attention. If you are ever in need of specialist legal advice regarding driving offences, or a range of other legal matters, you could speak to George Ide Solicitors, an independent full service law firm in London and Sussex.
Guide dogs have a special status within the pet world. They ensure that their owners have additional confidence when going out, but they also offer up the opportunity to travel further afield. There are many organisations that offer guidance on travelling abroad with pets and protecting them with appropriate pet insurance is key.
Prepare for departure
It’s essential that if you are planning to travel with a Guide or assistance dog, plan ahead and discuss travel plans with your chosen airport, airline, and booking agent as well as the local Guide Dog Mobility Team or relevant assistance dog organisation. Making sure that all personnel are aware of your dogs prescence, will reduce the possibility of complications on the day of travel.
A dogs “Mum or Dad” should ensure that they have appropriate identification for themselves and their dog, so a Pet Passport for international travel, and copies of any documentation provided by the airline. A working safety harness will be required too, which will secure the animal case, during the take off and landing. Make sure they have something familiar and comforting – such as an old woolen jumper and well used toy. Reducing their food intake for a few days prior to travelling will reduce the possibility of gastric problems on the flight.
Staff at the airport who deal specifically with Pets Passport documentation should have forwarded details of your situation onto the Animal Reception Centre. Once clearance has been obtained ? your dog is free to enjoy a fresh experience abroad.
A return trip will require a blood test which may result in treatment for tape worms ? so allow up to 120 hours before you have to board a plane. If your pet has picked up some unwanted guests, treatment can take anything from 24 ? 120 hours. There are essential pre booking arrangements required when travelling with a dog ? so ensure that you have completed all necessary tests in good time before your flight.
For a return journey to the UK an Animal Clearance Officer will meet the dog and owner on the plane or at the arrival gate. Paperwork is checked and then dog and owner are directed onto Customs and Immigration and baggage reclaim.
The PET Travel Scheme, is also now available for international and long haul flights. EC regulation 1107/2006 states that most UK airports must have facilities to enable Guide Dogs and their owners to travel safely ? and the UK is working towards that directive.
If you want to check out which airports and airlines are already accommodate guide and assistance dogs through the PET Travel Scheme, take a look at DEFRAs website www.defra.gov.uk .