A/B Testing for the Rest of Us

{title}

Update: A/B testing has been released and is available in all accounts.

This one has been hard to keep under wraps, but after loads of customer requests and a few months of hard work, I’m happy to announce that we’ll shortly be adding support for A/B testing to Campaign Monitor. This is one of those features that we always wanted to add, but were never completely satisfied with the best way to implement it.

Testing is one of the great strengths of email marketing. Almost everything is measurable, making it easy to compare multiple approaches and quickly figure out what works best for your subscribers. It’s a guesswork killer.

That’s the theory anyway. In reality, most of the A/B testing systems we’ve seen are bloated and hard to use beasts that require a manual just to get started. That never sat well with us, so we went back to the drawing board. We knew that if running an A/B test wasn’t as quick and easy as sending a regular campaign, it was never going to get used.

We plan on releasing this in the next month or so, but wanted to give you all the heads up on what’s coming. The testing feature will be available for everyone, including your clients if you’ve given them access to send their own campaigns.

Choose the test, we’ll do the rest

Instead of forcing you to create segments of lists, send multiple campaigns and scour over pages of statistics, our testing tool does all the heavy lifting for you.

All you need to do is choose what to test (subject line, from name or different email content) and how to decide the winner (open rate, link clicks, etc) and we’ll take it from there.

Campaign Monitor will then send both versions to a small subset of your recipients, see which version wins and automatically send that winning version to the remainder of your recipients. In a couple of clicks you’ve guaranteed that the best version of your email is being sent to your subscribers, and you’ve learned something doing it.

While we’ll be sharing more details when A/B testing goes live in the next month or so, here’s a sneak peek at how the process will look in your account.

A new type of campaign to send

When you create a new campaign, you’ll notice a new tab for running an A/B split campaign. Clicking that will let you choose what test you want to run and get started.

{title}

Setting the test parameters

A simple slider makes it easy to choose the size of your list you’d like to test on. Next choose how to pick the winner and how long the test should run for.

{title}

Watching a test in progress

For all you stats junkies out there, this sweet report gives you a bird’s eye view of a test as it’s running. When the winner is decided, we’ll send it to the rest of your list automatically. If you like, you can pre-empt this and send any time during the test.

{title}

Post-test report with benefit estimates

After the test is run and the winning version sent to the remainder of your recipients, you can access the full results of the test. We’ll also extrapolate the performance improvement of the winning version to give you an idea of the total benefit of sending that version to the remainder of your list. It’s a great way to justify testing to your client or boss.

{title}

When will this be available?

We’re putting the finishing touches on this feature now before running it through our usual phase of heavy testing (thanks to our new QA engineer Trish). While we can’t give an exact date just yet, we’re hoping to have this available in all accounts within a months time. We plan on sharing more as we get closer to launch.

If you have any feedback (positive or negative) or questions about this new feature, we’d love to hear them below.

Posted by David Greiner

57 Comments

  • Damien
    5th June

    Wow, that looks awesome - cant wait to play…

  • Matthew Weinberg
    5th June

    I think it’s very important that we can choose to *not* automatically send the “winner”, or to cancel it in the middle. It’s possible that neither does as well as we hoped, or, alternatively, that we just wanted to test to a very small group for now to use the results later.

    That said—this looks very, very cool and I’m excited to use it.

  • Dave Calleja
    5th June

    That. Is. Awesome.

  • Naca-Yoda
    5th June

    Very tidy! Well done!

  • Chris McMahon, Cincinnati
    5th June

    Awesome, thanks for giving me another benefit to sell.

  • David Greiner
    5th June

    Thanks guys, happy to hear you’ll find this useful.

    @Matthew, thanks for the suggestion. In an effort to keep things simple, you could just set the recipients of your campaign as a smaller segment of the entire list. That way, you’ll only be running the test on (and sending the winner to) a subset of your entire list.

    Of course, like many feature we’ve launched it will more than likely change and improve based on continued customer feedback. We’ll be starting simple.

  • Alex
    5th June

    Really nice implementation—I’m looking forward to putting this to the test, as it were.

    At risk of looking a gift horse in the mouth, is the option to test delivery time anywhere on your horizon?

  • David Greiner
    5th June

    @Alex, we’ve built things in a way that we can add additional test criteria if the demand is there, so it’s certainly something we’d consider. Thanks for the suggestion.

  • Finge
    5th June

    This will be a very nice addition indeed! Can’t wait. I love your approach and the fact that you are striving to keep it simple and clean!

    Keep up the good work!

  • Simeon Duncan
    5th June

    That is awesome. Can’t wait to have a play and have a test. Keep up the great work at Campaign Monitor!

  • Dimitris Rakopoulos
    5th June

    Great, can’t wait!

    Next on the To-Do list: Autoresponders + Triggers!

  • Alex van Ginneken
    6th June

    This is great functionality! Can’t wait!

  • Nippr
    6th June

    Great work. When delivery time gets added this is awesome.

    Suggestion would be to be able to test more than just two versions. Not that I think 20 would be very cool, I could find a use for three versions.

    Looking very much forward to this! Thanks.

  • Julian Wellings
    6th June

    Bring it on!  I like the way CM doesn’t just copy what everyone else is doing but looks hard at how to improve on it.

  • Joshua Clanton
    6th June

    Looks amazing, guys! I second Matthew’s suggestion that auto-sending the winner be optional, but not required. That way we can tweak further if the results aren’t quite what we were looking for.

  • Eric Oliver
    6th June

    LOVE it!
    I have only one feature request (and am seconding some of the other people above): Allow time of send to be one of the testing criteria.

    Most of my A/B testing in the past has been determining effective times to send to my list.

    Thanks again - very exciting!

  • Jason
    6th June

    Fantastic, and good timing! I was just contemplating the hassle of doing my own AB testing with the current template system (No duplicate previous template/campaign?)

    I can only second/third the above comments RE a manual ‘big’ send rather than fully automatic. Especially as this feature would also be really useful for reporting back to clients who may wish to do further changes, testing etc.

  • Smallfurrything
    6th June

    @Jason If you go to create/send there’s an option on the right to create a new campaign based on one you’ve sent. I can’t remember the exact wording.

    But this looks like a great addition CM, well done!

  • steven
    6th June

    This is a great idea. I cant wait to dive right into it!

  • Brad Haynes
    7th June

    Yeesh, you guys are so smart ;)

  • Simon
    8th June

    Fantastic stuff. Can’t wait to try it out.

    My addition to the wish list echoes the calls above for delivery time to be one of the criteria. To be able to show clients that a campaign achieves a better open rate at 9am versus 5pm (or not as it may be) would be a great help.

  • Anna Yeaman
    8th June

    This was no.1 on my wish list so I couldn’t be happier! Can’t wait to start using this.

    Agree with all the above suggestions about being able to test delivery time. This comes up quite often with my clients who seem obsessed with best time to send….

  • James
    8th June

    Hi guys… great news!
    Will the Google Analytics tags for the campaign name etc follow through with whether it was test A/B…? Again, useful in tracking conversions right the way through…

  • Simon
    9th June

    Wow, this is excellent!  Can’t wait to use it.

  • Al Stevens
    9th June

    Oh, just so so marvellous. Do we know when it’ll be ready!

  • Restoration Media
    9th June

    Awesome! This is truly awesome. I can’t wait to be able to try this out.

  • David Greiner
    9th June

    Thanks for all the requests for the ability to test best time to send, we’ve definitely taken that on board.

    @James - Yep, we’ll be appending a version number to the Google Analytics tags so you can track each version separately in your GA account.

    @Al, we’re hoping to have this released some time in the next month. We’ll be sharing more here as we get closer.

  • John Driscoll
    11th June

    No more guessing.  I love it!!

  • Brandon Eley
    11th June

    Wow, awesome! Can’t wait to try it out.

  • Ryan
    11th June

    Why manually choose the size of the test subset?

    Maybe the better way would be to send out until a statistically significant result was obtained (which could be just 10 if one design sucked or could be 10,000 emails if they are close) and then automatically or manually send the winning design to the rest.

  • Morgan
    11th June

    Killer feature, and dead simple to use. Best UI ever.

    Are you guys thinking of Salesforce integration, dynamic content or drip feed campaigns?

  • Jon S
    11th June

    Sweet UI, a welcome addition to the product.

  • Julia@Niche
    11th June

    Have I ever told you guys how much I love you?  Your consistent innovation ensures that Campaign Monitor is a quality product that works as well for us as it does for our clients. 

    Also, when will Lunch 2.0 be released to the public?  Seriously.

  • Basti
    12th June

    How much will it be? ;-)

  • Basti
    12th June

    Also agree with Matthew regarding the option not to send the winner automatically. If the differences are more complex I can imagine that I would like check them manually and than decide.

  • Chriz
    12th June

    Nice. Looking forward to see this in action

  • Rob Jones
    12th June

    I’ll second @Dimitris Rakopoulos for Autoresponders and most importantly Triggers!

    Also, would love to see A/B testing on delivery time as also mentioned above several times…

  • Elaine Nelson
    13th June

    I’m so excited about this! A/B testing has always been of a brainbender, and this should really help!

  • Jon M
    14th June

    This is REALLY impressive. A genuinely useful, added value feature with simplicity and focus. Look forward to receiving the launch email!

  • Jeff
    19th June

    Absolutely awesome.

    Quick Q: What kind of pricing model will be applied to this new feature?

  • David Greiner
    19th June

    Jeff, sending an A/B campaign will be just like sending a regular campaign. Even though we’re effectively sending 3 different campaigns, you only pay the usual $5 delivery and 1 cent/recipient.

  • makonja
    21st June

    I’m drooling about this feature. can’t wait to use it. will ease my marketing efforts a lot. Please make it fast :) thanks guys

  • makonja
    27th June

    we are daying for this A/B testing. do you have an approx eta?

  • Charles Garrison
    27th June

    Very cool!  I’ve been waiting for this feature since we started using Campaign Monitor :)

  • makonja
    2nd July

    any ETA yet?

  • Jon Moss
    5th July

    How’s the ETA looking chaps?!
    Yours excitedly,

    Jon :)

  • Rob Jones
    6th July

    Yup I’m still keen to know how that elusive ETA is coming along…  Any ideas please?

    Rob.

  • David Greiner
    6th July

    Hey guys, testing is coming along nice and we’re on the home stretch. We want to ensure the entire process runs like a dream, and we’re almost there. A release don’t won’t be more than a week or two away, and we’ll make an announcement in the blog as soon as it’s confirmed so you have plenty of notice.

    Great to see you all sharing our excitement to get this out in the wild.

  • Mike Bowzeylo
    7th July

    Yay…I can’t wait. This is going to be a stellar feature.

  • Chris
    16th July

    I’m glad i saw this as it was one of the things keeping a client off CM.  Hope it comes soon!

  • Mak
    6th August

    still waiting guys. whats up?

  • Mikkel Steen Andersen
    12th August

    When oh! when…?

  • Paul Mycroft
    13th August

    Not only will this help the sender in so many ways but it will hopefully reduce the number of client spam flags.

    Great job guys!

  • Matt Lerner
    15th August

    When is this feature launching?  My boss wants me to switch to an email service with A/B testing—so I hope you add it before we have to make the switch!

  • David Greiner
    15th August

    Thanks for your patience on this guys, we were all set for release before the hacking incident happened, so we’ll be resolving those issues as a matter of top priority before releasing this feature. Rest assured, it’s ready to roll, so as soon as we’re comfortable we’ll be releasing it.

    We’ll have more to share on this early next week. Thanks so much for your patience and understanding.

  • Matt
    17th August

    I have a couple of suggestions:

    1) statistics based selection of test recipients. I would like it if we could select from our pool of test subjects a percentage of subjects that meet certain criteria. 10% use outlook, 10% use mail.app etc perhaps track how many times a user opens their emails and then look at testing percentages on that sort of frequency.

    2) ability to exclude the original test subjects and run a second test. Obviously you don;t want to wait a week or custom duration until the next test. it would be good to be able to adjust the test and resend to a different test group.

    3) access to this functionality via the API :)

  • Lance
    3rd September

    Sept 02… turning blue ... can’t hold breath much ... longer… passing out.  Help me!

Sign up for free.
Then send campaigns for as little as $9/month

Create an account