A/B Testing is now available
It’s been a long time coming, but today I’m excited to announce that our new A/B testing feature we announced a while back is now live and available in all accounts.
Like we mentioned in the original announcement, we’ve worked hard to make sending an A/B campaign just as easy as sending a regular one. You can test different subject lines or sender details in a matter of seconds. If you have the time to create a simple design variation for your campaigns, this is also a breeze to test. I really can’t think of a good reason to send a regular old campaign any more.
The best part, you learn something from every single test you run.
One of my favorite new features that come with the A/B testing update are benefit estimates. Basically, after the test is run and the winning version is sent to the remainder of your subscribers, we’ll look at how much better the winning version performed in your test and extrapolate that across the rest of your campaign results. Check out the screenshot below for a campaign that received an additional 819 clicks by sending the winning version.
Benefit estimates give you a great indication on the total benefit gained from running the test. It’s also a great way to justify testing to your clients or the powers that be.
Now that A/B testing is out, we can roll out a host of other sweet new features we’ve been hard at work on. To help you get the most out of A/B testing, we’ll also be starting a new blog series next week with loads of info on how to plan a test, what you can test and different approaches you and your clients might want to try.
17th September
Very sweet tool and super slick implementation. You’ve not only done all the hard work…you’ve done _all_ the work. We just get to basically sit back and enjoy all the statistic goodness. Good job guys.
17th September
Great feature! I can’t wait to test it out on our newsletter.
17th September
Should be handy.
17th September
Is there any way to turn the feature off for specific clients?
(sorry for the split post!)
17th September
@Mike and @Aaron Thanks so much guys, looking forward to seeing what you do with it.
@g Not right now no, this is available for all clients that you’ve given access to create and send their own campaigns. If there is the demand though, we can certainly add this is an optional piece of functionality you can toggle on and off in client settings.
Anyone else got thoughts on this?
17th September
Awesome new feature CM, this is going to really help fine tune our campaigns.
17th September
I’d be curious to no the baseline variance if you sent the exact same message to both A and B? It should theoretically be zero but I sort of doubt it is.
17th September
Good call Derek, only one way to find out I guess. If the list size was big enough, there really shouldn’t be much deviation.
17th September
Congratulations with this big breakthru!
Excitiing to go and test this out…first have to make my template inserted in the campaing monitor envoirement.
17th September
are you able to test for different times of the day as well as different subject lines?
17th September
Great functionality, I’m going to test this right away!
17th September
@Paula The ability to test different times of day was highly requested when we announced the feature, and it’s definitely something we plan on supporting soon. Right now you can test on subject, sender details and different email content.
18th September
@David - Yes, we would really like this to be optional. We have a wide range of abilities across our clients. Some will want to use this themselves, others will want us to do it for them.
18th September
@Phil great, thanks for confirming that. We’ve got some nice improvements to the client permissions screen coming soon, and I’ve added a vote for the ability to set A/B tests as optional for clients.
The more people we hear this from, the sooner it will likely happen.
18th September
This is awesome. Additionally, I would just like to chime in and second Paula on the time of day testing—that’s been the most crucial testing we’ve done historically.
Thanks again!
18th September
Such awesome timing. My email best practices presentation next week can now include this important feature. Thank you!
18th September
very cool - will this be accessible via the API?
18th September
At last! Thank you! Really nice feature!
18th September
Glad to see the functionality. Looks good.
I see where you declare the winner. Do you also calculate & show the confidence level that the long term response rate differences will sustain as currently shown?. SplitTester.com does something comparable for PPC advertising.
18th September
To back up Terry, I’m also wondering if, amongst all the other fantastic things CM is working on, a statistical significance calculator could be thrown in so that we can see if the difference between A & B is enough to be relevant.
Thanks so much though, for this. Objectivity needs to be king in the professional world, and features like this really put shine on its crown.
18th September
Nice job. I want to add my vote for:
being able to turn it on and off for clients,
testing time of day,
and API access.
Great so far! Now make it completely irresistible.
18th September
I was very much looking forward to this feature as we are constantly battling over our subject lines. Will have fun with it I’m sure.
19th September
Another vote for the ability to turn on and off for clients. Thanks for a superb feature!
19th September
Voting for API access
19th September
Absolutely brilliant… I love it!
22nd September
Vote: API Access
23rd September
Would be good to turn on/off for clients.
Abit offtopic:
I work as a Webmaster for a large group. Im using campaignmonitor for all of the companies we own. I would like a feature to add subaccounts to have somewhat like this: superadmin (have all rights) - client admin (has the rights superadmin set to specific account) - client subuser (has the rights clientadmin set to clientadmins account) Some companies want to send themselves and have different roles in a company, someone is a marketing coordinator and someone is a Product Area Manager. Se what i mean?
24th September
@David Greiner Yes this shoul dbe optional as some of our clients have very small lists and this sin’t appropriate for them. Some clients may stop using this if they can’t switch it off.
Otherwise it’s a veyr usefull feature and well implemented so thanks CM!
24th September
How do I include a photo when creating a campaign ?
24th September
Al, if you can code your own HTML, you just do this on your own end and then import the complete campaign into Campaign Monitor.
25th September
that’s one of my wish list and it’s awesome…pretty easy to use though
25th September
API access please, great job, guys
25th September
Votes for turn on and off for specific clients and A/b based on time of day.
27th September
Awesome - another reason I am glad for choosing CM. Next mail shot will definitely try this out.
30th September
Very useful feature. I had been waiting for this feature to come to campaign monitor.
3rd October
I’m looking forward to the new blog on the A/B testing. This will be a very useful feature and I’m looking forward to using it.
On another note: I would love to go to Australia next week for the conference, but will not be able to as it’s too far and too much money. Could you send me a shirt?
16th October
Cool, will be very useful for us.
Only one thing though, why can I only send each version to 25%+25% to find a winner.
We would like to test a few functions on the whole group 50%-50% and then analyze the results to learn lessons for the next letters.
2nd November
Fantastic tool, thanks for improving the service!
2nd November
Our pleasure. We have some great guides in the blog now, as well - http://www.campaignmonitor.com/blog
If you have any suggestions as to how we can make A/B testing even better, don’t hesitate to drop us a line.