{title}Update: A/B testing has been released and is available in all accounts.

This one has been hard to keep under wraps, but after loads of customer requests and a few months of hard work, I’m happy to announce that we’ll shortly be adding support for A/B testing to Campaign Monitor. This is one of those features that we always wanted to add, but were never completely satisfied with the best way to implement it.

Testing is one of the great strengths of email marketing. Almost everything is measurable, making it easy to compare multiple approaches and quickly figure out what works best for your subscribers. It’s a guesswork killer.

That’s the theory anyway. In reality, most of the A/B testing systems we’ve seen are bloated and hard to use beasts that require a manual just to get started. That never sat well with us, so we went back to the drawing board. We knew that if running an A/B test wasn’t as quick and easy as sending a regular campaign, it was never going to get used.

We plan on releasing this in the next month or so, but wanted to give you all the heads up on what’s coming. The testing feature will be available for everyone, including your clients if you’ve given them access to send their own campaigns.

Choose the test, we’ll do the rest

Instead of forcing you to create segments of lists, send multiple campaigns and scour over pages of statistics, our testing tool does all the heavy lifting for you.

All you need to do is choose what to test (subject line, from name or different email content) and how to decide the winner (open rate, link clicks, etc) and we’ll take it from there.

Campaign Monitor will then send both versions to a small subset of your recipients, see which version wins and automatically send that winning version to the remainder of your recipients. In a couple of clicks you’ve guaranteed that the best version of your email is being sent to your subscribers, and you’ve learned something doing it.

While we’ll be sharing more details when A/B testing goes live in the next month or so, here’s a sneak peek at how the process will look in your account.

A new type of campaign to send

When you create a new campaign, you’ll notice a new tab for running an A/B split campaign. Clicking that will let you choose what test you want to run and get started.


Setting the test parameters

A simple slider makes it easy to choose the size of your list you’d like to test on. Next choose how to pick the winner and how long the test should run for.


Watching a test in progress

For all you stats junkies out there, this sweet report gives you a bird’s eye view of a test as it’s running. When the winner is decided, we’ll send it to the rest of your list automatically. If you like, you can pre-empt this and send any time during the test.


Post-test report with benefit estimates

After the test is run and the winning version sent to the remainder of your recipients, you can access the full results of the test. We’ll also extrapolate the performance improvement of the winning version to give you an idea of the total benefit of sending that version to the remainder of your list. It’s a great way to justify testing to your client or boss.


When will this be available?

We’re putting the finishing touches on this feature now before running it through our usual phase of heavy testing (thanks to our new QA engineer Trish). While we can’t give an exact date just yet, we’re hoping to have this available in all accounts within a months time. We plan on sharing more as we get closer to launch.

If you have any feedback (positive or negative) or questions about this new feature, we’d love to hear them below.

  • http://www.surefiredigital.co.uk Rob Jones

    Yup I’m still keen to know how that elusive ETA is coming along… Any ideas please?


  • http://www.campaignmonitor.com/ David Greiner

    Hey guys, testing is coming along nice and we’re on the home stretch. We want to ensure the entire process runs like a dream, and we’re almost there. A release don’t won’t be more than a week or two away, and we’ll make an announcement in the blog as soon as it’s confirmed so you have plenty of notice.

    Great to see you all sharing our excitement to get this out in the wild.

  • http://hdlive09.co.uk Jon Moss

    How’s the ETA looking chaps?!
    Yours excitedly,

    Jon :)

  • http://www.base2studio.com Mike Bowzeylo

    Yay…I can’t wait. This is going to be a stellar feature.

  • http://omipeople.com Mikkel Steen Andersen

    When oh! when…?

  • http://www.paulmycroft.com Paul Mycroft

    Not only will this help the sender in so many ways but it will hopefully reduce the number of client spam flags.

    Great job guys!

  • http://www.walkscore.com Matt Lerner

    When is this feature launching? My boss wants me to switch to an email service with A/B testing–so I hope you add it before we have to make the switch!

  • http://www.merlin-wizard.co.uk Jeff

    Absolutely awesome.

    Quick Q: What kind of pricing model will be applied to this new feature?

  • http://www.campaignmonitor.com/ David Greiner

    Jeff, sending an A/B campaign will be just like sending a regular campaign. Even though we’re effectively sending 3 different campaigns, you only pay the usual $5 delivery and 1 cent/recipient.

  • http://www.propellerglobal.com Damien

    Wow, that looks awesome – cant wait to play…

  • http://www.vectormediagroup.com Matthew Weinberg

    I think it’s very important that we can choose to *not* automatically send the “winner”, or to cancel it in the middle. It’s possible that neither does as well as we hoped, or, alternatively, that we just wanted to test to a very small group for now to use the results later.

    That said– this looks very, very cool and I’m excited to use it.

  • Dave Calleja

    That. Is. Awesome.

  • Naca-Yoda

    Very tidy! Well done!

  • http://twitter.com/cinciWebDesign Chris McMahon, Cincinnati

    Awesome, thanks for giving me another benefit to sell.

  • http://www.campaignmonitor.com/ David Greiner

    Thanks guys, happy to hear you’ll find this useful.

    @Matthew, thanks for the suggestion. In an effort to keep things simple, you could just set the recipients of your campaign as a smaller segment of the entire list. That way, you’ll only be running the test on (and sending the winner to) a subset of your entire list.

    Of course, like many feature we’ve launched it will more than likely change and improve based on continued customer feedback. We’ll be starting simple.

  • Morgan

    Killer feature, and dead simple to use. Best UI ever.

    Are you guys thinking of Salesforce integration, dynamic content or drip feed campaigns?

  • http://blogbitesman.net Jon S

    Sweet UI, a welcome addition to the product.

  • http://www.nichestudio.com.au Julia@Niche

    Have I ever told you guys how much I love you? Your consistent innovation ensures that Campaign Monitor is a quality product that works as well for us as it does for our clients.

    Also, when will Lunch 2.0 be released to the public? Seriously.

  • http://www.campaignmonitor.com/ David Greiner

    Thanks for all the requests for the ability to test best time to send, we’ve definitely taken that on board.

    @James – Yep, we’ll be appending a version number to the Google Analytics tags so you can track each version separately in your GA account.

    @Al, we’re hoping to have this released some time in the next month. We’ll be sharing more here as we get closer.

  • http://www.restorationmedia.com Restoration Media

    Awesome! This is truly awesome. I can’t wait to be able to try this out.

  • http://anderbose.com Brad Haynes

    Yeesh, you guys are so smart ;)

  • http://www.kitsimons.com Simon

    Fantastic stuff. Can’t wait to try it out.

    My addition to the wish list echoes the calls above for delivery time to be one of the criteria. To be able to show clients that a campaign achieves a better open rate at 9am versus 5pm (or not as it may be) would be a great help.

  • http://www.stylecampaign.com/ Anna Yeaman

    This was no.1 on my wish list so I couldn’t be happier! Can’t wait to start using this.

    Agree with all the above suggestions about being able to test delivery time. This comes up quite often with my clients who seem obsessed with best time to send….

  • Chriz

    Nice. Looking forward to see this in action

  • http://makonja.blogspot.com makonja

    I’m drooling about this feature. can’t wait to use it. will ease my marketing efforts a lot. Please make it fast :) thanks guys

  • http://www.surefiredigital.co.uk Rob Jones

    I’ll second @Dimitris Rakopoulos for Autoresponders and most importantly Triggers!

    Also, would love to see A/B testing on delivery time as also mentioned above several times…

  • James

    Hi guys… great news!
    Will the Google Analytics tags for the campaign name etc follow through with whether it was test A/B…? Again, useful in tracking conversions right the way through…

  • Simon

    Wow, this is excellent! Can’t wait to use it.

  • http://flukelondon.co.uk Al Stevens

    Oh, just so so marvellous. Do we know when it’ll be ready!

  • http://dialect.ca/ Alex

    Really nice implementation — I’m looking forward to putting this to the test, as it were.

    At risk of looking a gift horse in the mouth, is the option to test delivery time anywhere on your horizon?

  • http://www.campaignmonitor.com/ David Greiner

    @Alex, we’ve built things in a way that we can add additional test criteria if the demand is there, so it’s certainly something we’d consider. Thanks for the suggestion.

  • Finge

    This will be a very nice addition indeed! Can’t wait. I love your approach and the fact that you are striving to keep it simple and clean!

    Keep up the good work!

  • http://www.swordcdc.com/blog Simeon Duncan

    That is awesome. Can’t wait to have a play and have a test. Keep up the great work at Campaign Monitor!

  • http://www.redirect.gr Dimitris Rakopoulos

    Great, can’t wait!

    Next on the To-Do list: Autoresponders + Triggers!

  • http://www.pentalogic.net Ryan

    Why manually choose the size of the test subset?

    Maybe the better way would be to send out until a statistically significant result was obtained (which could be just 10 if one design sucked or could be 10,000 emails if they are close) and then automatically or manually send the winning design to the rest.

  • http://www.deblauwedoos.nl Alex van Ginneken

    This is great functionality! Can’t wait!

  • http://www.nippr.dk Nippr

    Great work. When delivery time gets added this is awesome.

    Suggestion would be to be able to test more than just two versions. Not that I think 20 would be very cool, I could find a use for three versions.

    Looking very much forward to this! Thanks.

  • http://www.expertiseontap.co.uk Julian Wellings

    Bring it on! I like the way CM doesn’t just copy what everyone else is doing but looks hard at how to improve on it.

  • http://www.designpepper.com Joshua Clanton

    Looks amazing, guys! I second Matthew’s suggestion that auto-sending the winner be optional, but not required. That way we can tweak further if the results aren’t quite what we were looking for.

  • http://www.thecosmonaut.com Eric Oliver

    LOVE it!
    I have only one feature request (and am seconding some of the other people above): Allow time of send to be one of the testing criteria.

    Most of my A/B testing in the past has been determining effective times to send to my list.

    Thanks again – very exciting!

  • http://www.custardfish.com Jason

    Fantastic, and good timing! I was just contemplating the hassle of doing my own AB testing with the current template system (No duplicate previous template/campaign?)

    I can only second/third the above comments RE a manual ‘big’ send rather than fully automatic. Especially as this feature would also be really useful for reporting back to clients who may wish to do further changes, testing etc.

  • Smallfurrything

    @Jason If you go to create/send there’s an option on the right to create a new campaign based on one you’ve sent. I can’t remember the exact wording.

    But this looks like a great addition CM, well done!

  • steven

    This is a great idea. I cant wait to dive right into it!

  • http://seedlng.com John Driscoll

    No more guessing. I love it!!

  • http://www.brandoneley.com Brandon Eley

    Wow, awesome! Can’t wait to try it out.

  • Basti

    How much will it be? ;-)

  • Basti

    Also agree with Matthew regarding the option not to send the winner automatically. If the differences are more complex I can imagine that I would like check them manually and than decide.

  • http://elainenelson.org Elaine Nelson

    I’m so excited about this! A/B testing has always been of a brainbender, and this should really help!

  • http://www.hulldigital.co.uk Jon M

    This is REALLY impressive. A genuinely useful, added value feature with simplicity and focus. Look forward to receiving the launch email!

  • http://makonja.blogspot.com makonja

    any ETA yet?

  • Charles Garrison

    Very cool! I’ve been waiting for this feature since we started using Campaign Monitor :)

  • http://makonja.blogspot.com Mak

    still waiting guys. whats up?

  • http://www.crispeegrafix.com/ Chris

    I’m glad i saw this as it was one of the things keeping a client off CM. Hope it comes soon!

  • http://makonja.blogspot.com makonja

    we are daying for this A/B testing. do you have an approx eta?

  • http://www.campaignmonitor.com/ David Greiner

    Thanks for your patience on this guys, we were all set for release before the hacking incident happened, so we’ll be resolving those issues as a matter of top priority before releasing this feature. Rest assured, it’s ready to roll, so as soon as we’re comfortable we’ll be releasing it.

    We’ll have more to share on this early next week. Thanks so much for your patience and understanding.

  • Matt

    I have a couple of suggestions:

    1) statistics based selection of test recipients. I would like it if we could select from our pool of test subjects a percentage of subjects that meet certain criteria. 10% use outlook, 10% use mail.app etc perhaps track how many times a user opens their emails and then look at testing percentages on that sort of frequency.

    2) ability to exclude the original test subjects and run a second test. Obviously you don;t want to wait a week or custom duration until the next test. it would be good to be able to adjust the test and resend to a different test group.

    3) access to this functionality via the API :)

  • Lance

    Sept 02… turning blue … can’t hold breath much … longer… passing out. Help me!

Want to improve your email marketing? Subscribe to get tips on improving your email marketing delivered to your inbox.

Join 150,000 companies around the world that use Campaign Monitor to run email marketing campaigns that deliver results for their business.

Get started for free