Sometimes, when running an A/B test, it may feel like "no one is winning" - neither the A nor the B variant is declared as the clear winner. This doesnt necessarily mean the test has failed. In fact, it often happens for a few reasons:
1. Both variants perform equally well
If the results of variant A and variant B are very close, the difference between them may not reach statistical significance. In that case, the system cannot confidently declare one as the winner.
2. How a significant winner is calculated ✅
We determine the winner based on conversion rate (CR):
Each test variant must have at least 10 orders to be considered.
The difference between variants is calculated using a statistical significance formula, specifically a z-test, to determine whether one variant performs better than the other in a meaningful way.
3. Not enough data yet
If the test hasn't run long enough or hasn't collected enough samples, it may remain undecided.
4. High performance overall is still a win
Even if there' no "significant winner", the fact that both variants perform well is a positive outcome. It shows that your base idea works, and you can use either variant confidently.
5. Next steps you can take
Let the test run longer to collect more data.
Consider testing a more impactful change to increase the chance of a measurable difference.
If both variants perform strongly, you may choose the one that best aligns with your brand or long-term goals.
Need help? Drop us a message in the live chat, and we'll sort it out together.