How to Get Useful Data From Losing and Inconclusive A/B Tests

Look Beyond Your Core Metrics

You might have hidden data in your losing test results.

E.g. Look at your “losing” ad to see if it drove less traffic but had higher conversions. If you’d only been looking at traffic & revenue, you might not have noticed this ad works better statistically, if not in rough numbers.

Dig into that data to find out why it drove less traffic & use that to improve your next set.

Remove Junk Data

Test results can be skewed by junk data. Remove that to see trends more clearly & drill down to find crucial trends. Remove:

• bot traffic.
• your traffic.
• competitor traffic.

Look for Biases & Get Rid of Them

Look for factors that could have impacted your test.

• Did you run a promotion?
• Was it during a traditionally busy/slow season in your industry?
• Did a competitor’s launch impact your tests?

Try to separate your results from those impacts. If you can’t, try rerunning the test.

Look at how your test was run. Did you randomize who saw which versions? Were both versions mobile-optimized?

Run Your A/B Tests Again

The goal is to continuously improve performance. Once you’ve completed a test & determined a winner (or determined there was none), test again.