A/B Testing is now available

It’s been a long time coming, but today I’m excited to announce that our new A/B testing feature we announced a while back is now live and available in all accounts.

{title}

Like we mentioned in the original announcement, we’ve worked hard to make sending an A/B campaign just as easy as sending a regular one. You can test different subject lines or sender details in a matter of seconds. If you have the time to create a simple design variation for your campaigns, this is also a breeze to test. I really can’t think of a good reason to send a regular old campaign any more.

The best part, you learn something from every single test you run.

One of my favorite new features that come with the A/B testing update are benefit estimates. Basically, after the test is run and the winning version is sent to the remainder of your subscribers, we’ll look at how much better the winning version performed in your test and extrapolate that across the rest of your campaign results. Check out the screenshot below for a campaign that received an additional 819 clicks by sending the winning version.

{title}

Benefit estimates give you a great indication on the total benefit gained from running the test. It’s also a great way to justify testing to your clients or the powers that be.

Now that A/B testing is out, we can roll out a host of other sweet new features we’ve been hard at work on. To help you get the most out of A/B testing, we’ll also be starting a new blog series next week with loads of info on how to plan a test, what you can test and different approaches you and your clients might want to try.

Posted by David Greiner

39 Comments

  • Mike Bowzeylo
    17th September

    Very sweet tool and super slick implementation. You’ve not only done all the hard work…you’ve done _all_ the work. We just get to basically sit back and enjoy all the statistic goodness. Good job guys.

  • Aaron
    17th September

    Great feature! I can’t wait to test it out on our newsletter.

  • g
    17th September

    Should be handy.

  • g
    17th September

    Is there any way to turn the feature off for specific clients?

    (sorry for the split post!)

  • David Greiner
    17th September

    @Mike and @Aaron Thanks so much guys, looking forward to seeing what you do with it.

    @g Not right now no, this is available for all clients that you’ve given access to create and send their own campaigns. If there is the demand though, we can certainly add this is an optional piece of functionality you can toggle on and off in client settings.

    Anyone else got thoughts on this?

  • David Bell
    17th September

    Awesome new feature CM, this is going to really help fine tune our campaigns.

  • Derek
    17th September

    I’d be curious to no the baseline variance if you sent the exact same message to both A and B? It should theoretically be zero but I sort of doubt it is.

  • David Greiner
    17th September

    Good call Derek, only one way to find out I guess. If the list size was big enough, there really shouldn’t be much deviation.

  • Bjorn van der Neut
    17th September

    Congratulations with this big breakthru!

    Excitiing to go and test this out…first have to make my template inserted in the campaing monitor envoirement.

  • Paula
    17th September

    are you able to test for different times of the day as well as different subject lines?

  • Jop Berkhout
    17th September

    Great functionality, I’m going to test this right away!

  • David Greiner
    17th September

    @Paula The ability to test different times of day was highly requested when we announced the feature, and it’s definitely something we plan on supporting soon. Right now you can test on subject, sender details and different email content.

  • Phil
    18th September

    @David - Yes, we would really like this to be optional. We have a wide range of abilities across our clients. Some will want to use this themselves, others will want us to do it for them.

  • David Greiner
    18th September

    @Phil great, thanks for confirming that. We’ve got some nice improvements to the client permissions screen coming soon, and I’ve added a vote for the ability to set A/B tests as optional for clients.

    The more people we hear this from, the sooner it will likely happen.

  • Eric Oliver
    18th September

    This is awesome. Additionally, I would just like to chime in and second Paula on the time of day testing—that’s been the most crucial testing we’ve done historically.

    Thanks again!

  • Brace Rosso
    18th September

    Such awesome timing.  My email best practices presentation next week can now include this important feature.  Thank you!

  • Tom Davies
    18th September

    very cool - will this be accessible via the API?

  • Tobias
    18th September

    At last! Thank you! Really nice feature!

  • Terry Brandon
    18th September

    Glad to see the functionality.  Looks good.

    I see where you declare the winner.  Do you also calculate & show the confidence level that the long term response rate differences will sustain as currently shown?.  SplitTester.com does something comparable for PPC advertising.

  • Sean Wood
    18th September

    To back up Terry, I’m also wondering if, amongst all the other fantastic things CM is working on, a statistical significance calculator could be thrown in so that we can see if the difference between A & B is enough to be relevant.

    Thanks so much though, for this. Objectivity needs to be king in the professional world, and features like this really put shine on its crown.

  • zed
    18th September

    Nice job. I want to add my vote for:
    being able to turn it on and off for clients, 
    testing time of day,
    and API access.
    Great so far! Now make it completely irresistible.

  • Jonathan Saipe
    18th September

    I was very much looking forward to this feature as we are constantly battling over our subject lines. Will have fun with it I’m sure.

  • Eamonn Dunne
    19th September

    Another vote for the ability to turn on and off for clients. Thanks for a superb feature!

  • Lars Buur
    19th September

    Voting for API access

  • Robin Morris
    19th September

    Absolutely brilliant… I love it!

  • Jeremy Glover
    22nd September

    Vote: API Access

  • Ola
    23rd September

    Would be good to turn on/off for clients.

    Abit offtopic:
    I work as a Webmaster for a large group. Im using campaignmonitor for all of the companies we own. I would like a feature to add subaccounts to have somewhat like this: superadmin (have all rights) - client admin (has the rights superadmin set to specific account) - client subuser (has the rights clientadmin set to clientadmins account) Some companies want to send themselves and have different roles in a company, someone is a marketing coordinator and someone is a Product Area Manager. Se what i mean?

  • smallfurrything
    24th September

    @David Greiner Yes this shoul dbe optional as some of our clients have very small lists and this sin’t appropriate for them. Some clients may stop using this if they can’t switch it off.
    Otherwise it’s a veyr usefull feature and well implemented so thanks CM!

  • Al
    24th September

    How do I include a photo when creating a campaign ?

  • David Greiner
    24th September

    Al, if you can code your own HTML, you just do this on your own end and then import the complete campaign into Campaign Monitor.

  • ziki
    25th September

    that’s one of my wish list and it’s awesome…pretty easy to use though

  • Ziki
    25th September

    API access please, great job, guys

  • Mark Skeet
    25th September

    Votes for turn on and off for specific clients and A/b based on time of day.

  • mark
    27th September

    Awesome - another reason I am glad for choosing CM. Next mail shot will definitely try this out.

  • S Jain
    30th September

    Very useful feature. I had been waiting for this feature to come to campaign monitor.

  • Shirley Hemphill
    3rd October

    I’m looking forward to the new blog on the A/B testing.  This will be a very useful feature and I’m looking forward to using it.

    On another note:  I would love to go to Australia next week for the conference, but will not be able to as it’s too far and too much money.  Could you send me a shirt?

  • Staffan Landin
    16th October

    Cool, will be very useful for us.

    Only one thing though, why can I only send each version to 25%+25% to find a winner.

    We would like to test a few functions on the whole group 50%-50% and then analyze the results to learn lessons for the next letters.

  • Claire Diiorio
    2nd November

    Fantastic tool, thanks for improving the service!

  • Ros Hodgekiss
    2nd November

    Our pleasure. We have some great guides in the blog now, as well - http://www.campaignmonitor.com/blog

    If you have any suggestions as to how we can make A/B testing even better, don’t hesitate to drop us a line.

Sign up for free.
Then send campaigns for as little as $9/month

Create an account