Case study – LottieFiles
About Hippoc
Hippoc is a predictive A/B testing platform that uses AI and neuroscience to instantly produce A/B test winners — without needing to spend time or money rolling out a cumbersome A/B test.
Applying a real-life A/B test, we used the Hippoc platform to predict and validate the existing test results.
Real-life A/B Test – LottieFiles
As reported on the go-to A/B testing case study website, GuessTheTest, the original A/B test was run for a company called LottieFiles on an A/B testing platform called ABtesting.ai for 15 days with a total of 90,307 unique visitors.
The experiment was run to determine which AI-created copy would increase clicks on the main Call To Action (CTA) button, prompting users to “Get started for free” with the LottieFiles tool.
The study was set-up as a multi-armed bandit test. As such, traffic was initially split equally across the 5 variants; however, as a winner began to emerge, traffic was weighted and diverted to the best performing version.
The winner: Version B achieved a 16.23% conversion rate which equated to a 45.17% lift over the control at a 90% confidence level.
The Hippoc predictive A/B test experimentation
To validate this result, a quick test was run on Hippoc, comparing the control (Image 2) against the variant (Image 1).
With an overall impact score of 62%, Hippoc successfully predicted version B would outperform the control version which has an overall impact score of 60%.
How can this result be explained?
The small improvement in the overall impact score is mainly due to the better performance of the text in the version B compared to the control version.
By reducing the text, the brain is able to process the information more quickly which results in a slightly lower attention score in version B (71%) than in the control version (78%) but a higher recall score in version B.
The shorter text in version B also slightly improved the impact score on the CTA by slightly increasing the ability to attract attention (91% vs 90%) but also to be recall (80% vs 78%).
Assuming it costs LottieFiles $0.50 cents per visitor to lead 90,307 users to the 5 paid landing pages, this experiment cost over $45,000 in paid traffic and took up to two weeks to run.
In contrast, it took Hippoc less than 1 minute to run this test with the same winning results.
Learn more about this test and case study on GuessTheTest.