Over the years, conversion rate optimization (CRO) seems to have become synonymous with A/B testing in the minds of many marketers.A/B testing is a form of conversion rate optimization. You have a page and you want it to perform better, so you change something and see if it improves your results.But, A/B testing isn’t the only way to do CRO.If you’ve got enough traffic, multivariate testing can allow you to produce meaningful results much more quickly.

A/B Testing

Optimizely studied & reported on the factors that defined the world’s best testing companies. The 4 biggest factors were:<ol><li>Testing the things that drive the most revenue</li><li>Testing every change</li><li>Testing to solve real problems</li><li>Testing multiple variants simultaneously</li></ol>According to Optimizely, just 14% of A/B tests significantly improve conversion rates. On the other hand, tests with 4 variants improve conversion rates 27% of the time.So, if you test 4 variants, you are 90% more likely to improve your conversion rate than if you just ran an A/B test. However, 65% of CRO tests are A/B tests!

Multiple Variant Testing

With multivariate testing, you can try out several ideas at the same time. That means you can simultaneously test multiple hypotheses. This will allow you to optimize your page or site much more quickly than you could with a long series of A/B tests.Plus, running a multivariate test will greatly improve the odds that a single test will deliver at least one positive result, allowing you to start getting more from your website sooner.

Multivariate Testing is Faster

Things like differences in your competitors' marketing strategies, political change, or a variety of other variables can make it difficult to directly compare the results of A/B tests.As a result, sometimes it can be hard to know if a particular A/B testing variant succeeded (or failed) because of factors outside of your control or even knowledge. The more tests you run, the murkier your results may become.However, with a multivariate test, you are testing all of your variants under the same conditions. That makes it easy to compare results & draw valid, reliable conclusions from your tests.

Multivariate Testing is More Reliable

As great as multivariate testing is, if you don’t have enough traffic, a test could take months or years to complete.In fact, in true multivariate testing—where you test to see how a large number of subtle changes interact to generate your conversion rate—you want at least 100,000 unique visitors per month.On the other hand, you need far less traffic to simultaneously test multiple page variants.If the time frame makes sense for your business, go for it!

Should You Test Multiple Variants?

I hope you enjoy reading this blog post.If you want my team to just do your marketing for you, <a href="https://neilpatel.com/consulting/" dir="ltr">click here</a>.<h1>A/B Testing vs Multiple Variant Testing: And the Winner Is…?</h1><div><a href="https://neilpatel.com/blog/" dir="ltr">Blog</a> / <a href="https://neilpatel.com/blog/category/conversion-rate-optimization/" dir="ltr">Conversion Rate Optimization</a> / A/B Testing vs Multiple Variant Testing: And the Winner Is…?</div>During&nbsp;the 2016 Rio Olympic Games,&nbsp;<a href="http://bleacherreport.com/articles/2657500-olympic-rowing-2016-medal-winners-table-and-times-for-saturdays-results" rel="noopener" dir="ltr">Mahe Drysdale rowed 2,000 meters</a> (1.24 miles) in just 6 minutes and 41 seconds.However, despite his impressive&nbsp;performance,&nbsp;the world record-holder nearly lost the race.<img width="450" height="223" src="https://neilpatel.com/wp-content/uploads/2016/09/drysdale-rowing-win.gif" alt="Drysdale Rows to Nail-Biting Single Sculls Win in Rio 2016 Olympics">In one of the closest finishes in Olympic history, Drysdale won by mere millimeters.<div class="embed-placeholder" data-type="twitter" data-id="764457821085786113"></div>In contrast, Great Britain’s Men’s Eight team rowed the same distance in just 5 minutes and 29 seconds—over 70 seconds faster than Drysdale’s time!What’s more, the Brits won&nbsp;by more than a half second.<img width="450" height="253" src="https://neilpatel.com/wp-content/uploads/2016/09/gbr-mens-8-rowing-win.gif" alt="gbr-mens-8-rowing-win">That might not seem like a huge margin, but in the Olympics, a half second is a big deal.So, why was Britain’s team so much faster than Drysdale?The answer is simple: they had more oars in the water.Now, at this point, you might be thinking,&nbsp;This is all well and good, Jake, but what does rowing have to do with online marketing?Well,&nbsp;it turns out that conversion rate optimization (CRO) is a lot like rowing.The more oars you have in the water, the faster you’ll make it to your goal and the more likely you are to beat out the competition.<img src="https://neilpatel.com/wp-content/themes/neilpatel/images/blog/np2.png" width="262" height="338" alt="Consulting with Neil Patel">See How My Agency Can Drive Massive Amounts of Traffic to Your Website<UL><LI>SEO - unlock massive amounts of SEO traffic. See real results.</LI><LI>Content Marketing - our team creates epic content that will get shared, get links, and attract traffic.</LI><LI>Paid Media - effective paid strategies with clear ROI.</LI></UL><h2>The Secret is Testing Multiple Variants</h2>Over the years, CRO seems to have become synonymous with A/B testing in the minds of many&nbsp;marketers.Now, there’s nothing inherently wrong with this. A/B testing is a form of conversion rate optimization.&nbsp;You have a page and you want it to perform better, so you change something and see if it improves your results.But here’s the thing, A/B testing isn’t the&nbsp;only way to do CRO.It might not roll off the tongue as nicely as “A/B testing”, but if you’ve got enough traffic, A/B/C/D/etc testing can allow you to produce&nbsp;meaningful results much more quickly.For example, <a href="https://www.youtube.com/watch?v=BuGkE6FovbI" rel="noopener" dir="ltr">Optimizely recently studied and reported on the factors</a> that&nbsp;defined the world’s best testing companies.Guess what the 4 biggest factors were?<OL><LI>Testing&nbsp;the things that drive the most revenue</LI><LI>Testing&nbsp;every&nbsp;change</LI><LI>Testing to solve&nbsp;real problems</LI><LI>Testing multiple variants&nbsp;simultaneously</LI></OL>Does #4 surprise you?Apparently, the most effective CRO&nbsp;doesn’t come from&nbsp;A/B testing—it comes from testing multiple variants.Essentially,&nbsp;A/B testing is like the Mahe Drysdale of CRO. It works and it can even deliver amazing results.But, it’s only two oars in the water—there’s no way it can compete with an 8-man team.To put this in more concrete terms, according to Optimizely, just 14% of A/B tests&nbsp;significantly improve conversion rates. On the other hand, tests with 4&nbsp;variants improve conversion rates 27% of the time.So, if you test 4&nbsp;variants, you are 90% more likely to improve your conversion rate than if you just ran an A/B test.&nbsp;However, 65% of CRO&nbsp;tests are—you guessed it—A/B tests!<h2>Why Testing Multiple Variants Works Better</h2>Basically, there are two reasons why multiple variant testing outperforms A/B testing: 1) it’s faster and 2) it allows you to test more&nbsp;variants under the same testing conditions.<h3>Multiple Variant Testing is Faster</h3>Sure, you can test the same things with a series of A/B tests as you can with a multiple variant test—it just takes&nbsp;a lot&nbsp;longer.When you run an A/B test, you can really only learn one thing from your test. Your variant will either perform better, the same or worse than your original.And that’s it, that’s all you can learn.Now, if you’re smart about your A/B testing strategy, your results can teach you a lot about your audience and&nbsp;make your future tests smarter, but you’re still only learning one thing from each&nbsp;test.On the other hand, with multiple variant testing, you can try out several&nbsp;ideas&nbsp;at the same time. That means you can simultaneously test multiple hypotheses.So, instead of just learning that a hero shot with a smiling woman outperforms a shot of a grumpy man, you can also&nbsp;see&nbsp;if a grumpy woman image&nbsp;drives more results than the&nbsp;grumpy man pic or if a happy man outshines them all.Or, you can try multiple combinations, like a new headline or CTA in combination with either the smiling woman or the grumpy man.Running all of these tests simultaneously will allow you to optimize your page or site much more quickly than you could with a long series of A/B tests.Plus, running a test with multiple variants will greatly improve the odds that a single test will deliver at least one positive result, allowing you to start getting more from your website&nbsp;sooner.<h3>Multiple Variant&nbsp;Testing is More Reliable</h3>Another problem with&nbsp;successive A/B tests&nbsp;stems from the fact that the world changes over time.For example, if you are in eCommerce&nbsp;and run your first A/B test during October and your second test during November, how do you know if your results aren’t being skewed by Black Friday?Even if your business isn’t seasonal, things like differences in your competitors marketing strategies, political change or a variety of other variables can make it difficult to directly compare the results of A/B tests.As a result, sometimes it can be hard to know if&nbsp;a particular A/B testing variant succeeded (or failed) because of factors outside of your control or even knowledge. The more tests you run, the murkier your results may become.However, with a multiple variant test, you are testing all of your variants under the same conditions. That makes it easy to compare apples-to-apples and draw valid, reliable conclusions from your tests.<h3>What Does Testing Multiple Variants Look Like in Real Life?</h3>To show you just how testing multiple variants can improve your CRO results, let me share an experience we recently had with one of our clients.The client&nbsp;wanted to get site traffic to their “Find Your Local Chapter” page, so&nbsp;we decided to add a “Find Your Local Chapter” link to the client’s footer. That way, the link would be seen by as many people as possible.Makes sense, right?So, we put together something that looked like this:<img src="https://neilpatel.com/wp-content/uploads/2016/09/testing-multiple-footer-variants-v1.png" alt="testing-multiple-footer-variants-v1" width="450" height="450">At first, we figured we would just put the link in the footer and run a test to see if the link made a difference.But then, we started wondering if there was a way to make the link even more noticeable. After all, getting traffic to this page was a big deal to the client, so it made sense to emphasize the link.With that in mind, we added color to the link:<img src="https://neilpatel.com/wp-content/uploads/2016/09/testing-multiple-footer-variants-v2.png" alt="testing-multiple-footer-variants-v2" width="450" height="450">Now, this idea seemed logical, but at Disruptive, <a href="http://www.disruptiveadvertising.com/conversion-rate-optimization/ab-testing-sixth-sense/" rel="noopener" dir="ltr">we believe in testing, not gut instinct</a>, so we figured, “Hey, we’ve got enough traffic to test 3 variants, let’s take this even&nbsp;further!”The problem was, the client’s site was a designer’s dream—modern and seamlessly designed. To be honest, we had a bit of trouble selling them on the idea that creating a page element that interrupted their seamless flow was worth testing.But, eventually, we convinced them to try the following:<img src="https://neilpatel.com/wp-content/uploads/2016/09/testing-multiple-footer-variants-v3.png" alt="testing-multiple-footer-variants-v3" width="450" height="450">It was very different from anything the client had tried&nbsp;on the page before, but we decided to run with the idea and include it in our test.A few weeks and 110,000 visitors later, we had our winner: 
<img width="700" height="284" src="https://neilpatel.com/wp-content/uploads/2016/09/testing-multiple-footer-variants-results.png" alt="testing-multiple-footer-variants-results">Not surprisingly, adding the&nbsp;“Find Your Local Chapter” link&nbsp;increased page visits by over&nbsp;60% for every variant—that’s an awesome win, right?But here’s the thing. With our original, strict A/B test, we would only have discovered that adding the link increased traffic by 63%.On the other hand, by including a couple of extra variants, we were able with the same test to discover that—contrary to the client’s belief—the more our link “interrupted” the site experience, the more traffic it drove to the chapter page.Sure, we might have reached the same conclusion with several more tests, but we achieved these results much more quickly and reliably than we would have with an A/B testing series.<h2>Should You Test&nbsp;Multiple Variants?</h2>When it comes to testing multiple variants, there’s only one real reason not to use it: your boat is too small.<img width="300" height="169" src="https://neilpatel.com/wp-content/uploads/2016/09/rowing-fail.gif" alt="rowing-fail">Think about it: if the entire British Eight Man team had tried to cram onto Mahe Drysdale’s boat, they never would have made any forward progress.The same idea applies to CRO.As great as multiple variant&nbsp;testing is, if you don’t have enough traffic, a test could take months or years to complete.In fact, in true multivariate testing—where you test to see how a large number of subtle changes interact to generate&nbsp;your conversion rate—you want at least&nbsp;100,000 unique visitors per month (for more information on multivariate testing, <a href="http://conversionxl.com/multivariate-tests/" rel="noopener" dir="ltr">check out this great article</a>).On the other hand, you need far less traffic to simultaneously&nbsp;test multiple page variants.To see how long a multiple variant test will take on your site, try out this&nbsp;<a href="https://vwo.com/ab-split-test-duration/" rel="noopener" dir="ltr">VWO has a free sample size and test duration calculator from VWO</a>. If the time frame makes sense for your business, go for it!<h2>Conclusion</h2>Whether it’s Olympic rowing or CRO, the more oars you have in the water, the better your results will be.Although it may be tempting to limit CRO to A/B testing,&nbsp;testing multiple variants will allow you to improve your conversion rates more quickly and reliably than you could with a series of A/B tests.You’ve heard my two cents, now it’s your turn.Have you tried&nbsp;multiple variant testing?&nbsp;What was your experience like? Did any of the data in this article surprise you?About the Author: Jacob Baadsgaard is the CEO and fearless leader of <a href="https://bit.ly/1Y9YHM5" rel="noopener" dir="ltr">Disruptive Advertising</a>, an online marketing agency dedicated to using PPC advertising and website optimization to drive sales. His face is as big as his heart and he loves to help businesses achieve their online potential. Connect with him on <a href="https://www.linkedin.com/in/jakebaadsgaard" rel="noopener" dir="ltr">LinkedIn</a> or <a href="https://twitter.com/jakebaadsgaard" rel="noopener" dir="ltr">Twitter</a>.<h3>Share</h3><UL><LI><img src="https://neilpatel.com/wp-content/themes/neilpatel/images/icons/facebook-f-brands.svg" width="13" height="13" alt="Facebook"></LI><LI><img src="https://neilpatel.com/wp-content/themes/neilpatel/images/icons/twitter-brands.svg" width="21" height="21" alt="Twitter"></LI><LI><img src="https://neilpatel.com/wp-content/themes/neilpatel/images/icons/linkedin-in-brands.svg" width="18" height="18" alt="LinkedIn"></LI></UL>

A/B testing isn’t the only way to do CRO. It might not roll off the tongue as nicely as “A/B testing”, but if you’ve got enough traffic, A/B/C/D/etc testing can allow you to produce meaningful results much more quickly.

Discover the most interesting and off-beat ideas in marketing and advertising! 

Marketing & Advertising

I hope you enjoy reading this blog post.

If you want my team to just do your marketing for you, <a dir="ltr" href="https://neilpatel.com/consulting/">click here </a>.

A/B Testing vs Multiple Variant Testing: And the Winner Is…?

During the 2016 Rio Olympic Games, <a dir="ltr" href="http://bleacherreport.com/articles/2657500-olympic-rowing-2016-medal-winners-table-and-times-for-saturdays-results" rel="noopener">Mahe Drysdale rowed 2,000 meters </a>(1.24 miles) in just 6 minutes and 41 seconds.

However, despite his impressive performance, the world record-holder nearly lost the race.

In one of the closest finishes in Olympic history, Drysdale won by mere millimeters.

In contrast, Great Britain’s Men’s Eight team rowed the same distance in just 5 minutes and 29 seconds—over 70 seconds faster than Drysdale’s time!

What’s more, the Brits won by more than a half second.

That might not seem like a huge margin, but in the Olympics, a half second is a big deal.

So, why was Britain’s team so much faster than Drysdale?

The answer is simple: they had more oars in the water.

Now, at this point, you might be thinking, This is all well and good, Jake, but what does rowing have to do with online marketing?

Well, it turns out that conversion rate optimization (CRO) is a lot like rowing.

The more oars you have in the water, the faster you’ll make it to your goal and the more likely you are to beat out the competition.

See How My Agency Can Drive Massive Amounts of Traffic to Your Website

<ul><li>SEO - unlock massive amounts of SEO traffic. See real results. </li><li>Content Marketing - our team creates epic content that will get shared, get links, and attract traffic. </li><li>Paid Media - effective paid strategies with clear ROI. </li> </ul>

Over the years, CRO seems to have become synonymous with A/B testing in the minds of many marketers.

Now, there’s nothing inherently wrong with this. A/B testing is a form of conversion rate optimization. You have a page and you want it to perform better, so you change something and see if it improves your results.

But here’s the thing, A/B testing isn’t the only way to do CRO.

It might not roll off the tongue as nicely as “A/B testing”, but if you’ve got enough traffic, A/B/C/D/etc testing can allow you to produce meaningful results much more quickly.

For example, <a dir="ltr" href="https://www.youtube.com/watch?v=BuGkE6FovbI" rel="noopener">Optimizely recently studied and reported on the factors </a>that defined the world’s best testing companies.

Guess what the 4 biggest factors were?

<ol><li>Testing the things that drive the most revenue </li><li>Testing every  change </li><li>Testing to solve real problems </li><li>Testing multiple variants simultaneously </li> </ol>

Apparently, the most effective CRO doesn’t come from A/B testing—it comes from testing multiple variants.

Essentially, A/B testing is like the Mahe Drysdale of CRO. It works and it can even deliver amazing results.

But, it’s only two oars in the water—there’s no way it can compete with an 8-man team.

To put this in more concrete terms, according to Optimizely, just 14% of A/B tests significantly improve conversion rates. On the other hand, tests with 4 variants improve conversion rates 27% of the time.

So, if you test 4 variants, you are 90% more likely to improve your conversion rate than if you just ran an A/B test. However, 65% of CRO tests are—you guessed it—A/B tests!

Why Testing Multiple Variants Works Better

Basically, there are two reasons why multiple variant testing outperforms A/B testing: 1) it’s faster and 2) it allows you to test more variants under the same testing conditions.

Sure, you can test the same things with a series of A/B tests as you can with a multiple variant test—it just takes a lot  longer.

When you run an A/B test, you can really only learn one thing from your test. Your variant will either perform better, the same or worse than your original.

And that’s it, that’s all you can learn.

Now, if you’re smart about your A/B testing strategy, your results can teach you a lot about your audience and make your future tests smarter, but you’re still only learning one thing from each test.

On the other hand, with multiple variant testing, you can try out several ideas at the same time . That means you can simultaneously test multiple hypotheses.

So, instead of just learning that a hero shot with a smiling woman outperforms a shot of a grumpy man, you can also see if a grumpy woman image drives more results than the grumpy man pic or if a happy man outshines them all.

Or, you can try multiple combinations, like a new headline or CTA in combination with either the smiling woman or the grumpy man.

Running all of these tests simultaneously will allow you to optimize your page or site much more quickly than you could with a long series of A/B tests.

Plus, running a test with multiple variants will greatly improve the odds that a single test will deliver at least one positive result, allowing you to start getting more from your website sooner.

Multiple Variant Testing is More Reliable

Another problem with successive A/B tests stems from the fact that the world changes over time.

For example, if you are in eCommerce and run your first A/B test during October and your second test during November, how do you know if your results aren’t being skewed by Black Friday?

Even if your business isn’t seasonal, things like differences in your competitors marketing strategies, political change or a variety of other variables can make it difficult to directly compare the results of A/B tests.

As a result, sometimes it can be hard to know if a particular A/B testing variant succeeded (or failed) because of factors outside of your control or even knowledge. The more tests you run, the murkier your results may become.

However, with a multiple variant test, you are testing all of your variants under the same conditions. That makes it easy to compare apples-to-apples and draw valid, reliable conclusions from your tests.

What Does Testing Multiple Variants Look Like in Real Life?

To show you just how testing multiple variants can improve your CRO results, let me share an experience we recently had with one of our clients.

The client wanted to get site traffic to their “Find Your Local Chapter” page, so we decided to add a “Find Your Local Chapter” link to the client’s footer. That way, the link would be seen by as many people as possible.

So, we put together something that looked like this:

At first, we figured we would just put the link in the footer and run a test to see if the link made a difference.

But then, we started wondering if there was a way to make the link even more noticeable. After all, getting traffic to this page was a big deal to the client, so it made sense to emphasize the link.

With that in mind, we added color to the link:

Now, this idea seemed logical, but at Disruptive, <a dir="ltr" href="http://www.disruptiveadvertising.com/conversion-rate-optimization/ab-testing-sixth-sense/" rel="noopener">we believe in testing, not gut instinct </a>, so we figured, “Hey, we’ve got enough traffic to test 3 variants, let’s take this even further!”

The problem was, the client’s site was a designer’s dream—modern and seamlessly designed. To be honest, we had a bit of trouble selling them on the idea that creating a page element that interrupted their seamless flow was worth testing.

But, eventually, we convinced them to try the following:

It was very different from anything the client had tried on the page before, but we decided to run with the idea and include it in our test.

A few weeks and 110,000 visitors later, we had our winner:

Not surprisingly, adding the “Find Your Local Chapter” link increased page visits by over 60% for every variant—that’s an awesome win, right?

But here’s the thing. With our original, strict A/B test, we would only have discovered that adding the link increased traffic by 63%.

On the other hand, by including a couple of extra variants, we were able with the same test to discover that—contrary to the client’s belief—the more our link “interrupted” the site experience, the more traffic it drove to the chapter page.

Sure, we might have reached the same conclusion with several more tests, but we achieved these results much more quickly and reliably than we would have with an A/B testing series.

When it comes to testing multiple variants, there’s only one real reason not to use it: your boat is too small.

Think about it: if the entire British Eight Man team had tried to cram onto Mahe Drysdale’s boat, they never would have made any forward progress.

As great as multiple variant testing is, if you don’t have enough traffic, a test could take months or years to complete.

In fact, in true multivariate testing—where you test to see how a large number of subtle changes interact to generate your conversion rate—you want at least 100,000 unique visitors per month (for more information on multivariate testing, <a dir="ltr" href="http://conversionxl.com/multivariate-tests/" rel="noopener">check out this great article </a>).

On the other hand, you need far less traffic to simultaneously test multiple page variants.

To see how long a multiple variant test will take on your site, try out this <a dir="ltr" href="https://vwo.com/ab-split-test-duration/" rel="noopener">VWO has a free sample size and test duration calculator from VWO </a>. If the time frame makes sense for your business, go for it!

Whether it’s Olympic rowing or CRO, the more oars you have in the water, the better your results will be.

Although it may be tempting to limit CRO to A/B testing, testing multiple variants will allow you to improve your conversion rates more quickly and reliably than you could with a series of A/B tests.

You’ve heard my two cents, now it’s your turn.

Have you tried multiple variant testing? What was your experience like? Did any of the data in this article surprise you?

About the Author: Jacob Baadsgaard is the CEO and fearless leader of <a dir="ltr" href="https://bit.ly/1Y9YHM5" rel="noopener">Disruptive Advertising </a>, an online marketing agency dedicated to using PPC advertising and website optimization to drive sales. His face is as big as his heart and he loves to help businesses achieve their online potential. Connect with him on <a dir="ltr" href="https://www.linkedin.com/in/jakebaadsgaard" rel="noopener">LinkedIn </a>or <a dir="ltr" href="https://twitter.com/jakebaadsgaard" rel="noopener">Twitter </a>.

Bite-sized knowledge
to upgrade
your career

A/B Testing vs Multiple Variant Testing: And the Winner Is...?

A/B Testing

Multiple Variant Testing

Multivariate Testing is Faster

Multivariate Testing is More Reliable

Should You Test Multiple Variants?

IDEAS ABOUT

MORE LIKE THIS

A Beginner's Guide To A/B Testing: An Introduction

How to Get Useful Data From Losing and Inconclusive A/B Tests

What is an A/B Test?

It's time to

READ

LIKE

A PRO!

Bite-sized knowledgeto upgradeyour career

A/B Testing vs Multiple Variant Testing: And the Winner Is...?

A/B Testing

Multiple Variant Testing

Multivariate Testing is Faster

Multivariate Testing is More Reliable

Should You Test Multiple Variants?

IDEAS ABOUT

MORE LIKE THIS

A Beginner's Guide To A/B Testing: An Introduction

How to Get Useful Data From Losing and Inconclusive A/B Tests

What is an A/B Test?

It's time to

READ

LIKE

A PRO!

Bite-sized knowledge
to upgrade
your career