Campaign monitor provides a report that shows the performance of A/B subject line tests.
This report includes a stat and a graph on open rates.
When an A/B test is ended early this stat and graph should *only* report on opens that occurred before the test was manually stopped.
What I am seeing, unexpectedly is that opens are recorded for the full planned test period, even if that test period ended early.
The result is that whichever subject line was the winner has included in it's tally opens from emails that were send outside of the testing period.
I set up a 2 hour test. After one hour I see that A has 15% opens and B has 25% open. I decide to end the test early and send the B message.
Instead if I look after the full 2 hours has elapsed I see that A is at 16% and B is at 35%. This seems to be because the B version is getting incremented with the post-test emails.
I feel that the "final" score for the test should be 15 vs 25 and the graph should end at one hour. OR if you have a way to track B test emails against B non-test emails, you could extend the graph to 2 hours. In any case you must NOT count non-test emails as test emails.
Thanks very much for considering this report.
Hi phripley, a huge thanks for this bug report - I've noted all the details down and will get our dev team to look into why this is the case. We'll keep you updated as to whether a fix gets slipped into an upcoming release.
If you pop us a line with your shirt size and postal address, we'll happily send you a t-shirt for your excellent bug-spotting. Many thanks again for the detailed report!