Blog

A/B Testing for the Rest of Us

June 05, 2009

Update: A/B testing has been released and is available in all accounts.

This one has been hard to keep under wraps, but after loads of customer requests and a few months of hard work, I'm happy to announce that we'll shortly be adding support for A/B testing to Campaign Monitor. This is one of those features that we always wanted to add, but were never completely satisfied with the best way to implement it.

Testing is one of the great strengths of email marketing. Almost everything is measurable, making it easy to compare multiple approaches and quickly figure out what works best for your subscribers. It's a guesswork killer.

That's the theory anyway. In reality, most of the A/B testing systems we've seen are bloated and hard to use beasts that require a manual just to get started. That never sat well with us, so we went back to the drawing board. We knew that if running an A/B test wasn't as quick and easy as sending a regular campaign, it was never going to get used.

We plan on releasing this in the next month or so, but wanted to give you all the heads up on what's coming. The testing feature will be available for everyone, including your clients if you've given them access to send their own campaigns.

Choose the test, we'll do the rest

Instead of forcing you to create segments of lists, send multiple campaigns and scour over pages of statistics, our testing tool does all the heavy lifting for you.

All you need to do is choose what to test (subject line, from name or different email content) and how to decide the winner (open rate, link clicks, etc) and we'll take it from there.

Campaign Monitor will then send both versions to a small subset of your recipients, see which version wins and automatically send that winning version to the remainder of your recipients. In a couple of clicks you've guaranteed that the best version of your email is being sent to your subscribers, and you've learned something doing it.

While we'll be sharing more details when A/B testing goes live in the next month or so, here's a sneak peek at how the process will look in your account.

A new type of campaign to send

When you create a new campaign, you'll notice a new tab for running an A/B split campaign. Clicking that will let you choose what test you want to run and get started.

Setting the test parameters

A simple slider makes it easy to choose the size of your list you'd like to test on. Next choose how to pick the winner and how long the test should run for.

Watching a test in progress

For all you stats junkies out there, this sweet report gives you a bird's eye view of a test as it's running. When the winner is decided, we'll send it to the rest of your list automatically. If you like, you can pre-empt this and send any time during the test.

Post-test report with benefit estimates

After the test is run and the winning version sent to the remainder of your recipients, you can access the full results of the test. We'll also extrapolate the performance improvement of the winning version to give you an idea of the total benefit of sending that version to the remainder of your list. It's a great way to justify testing to your client or boss.

When will this be available?

We're putting the finishing touches on this feature now before running it through our usual phase of heavy testing (thanks to our new QA engineer Trish). While we can't give an exact date just yet, we're hoping to have this available in all accounts within a months time. We plan on sharing more as we get closer to launch.

If you have any feedback (positive or negative) or questions about this new feature, we'd love to hear them below.

Posted in: New Features & Updates

57 Comments

Damien
June 5, 2009 1:08pm

Wow, that looks awesome - cant wait to play…
Matthew Weinberg
June 5, 2009 1:33pm

I think it’s very important that we can choose to *not* automatically send the “winner”, or to cancel it in the middle. It’s possible that neither does as well as we hoped, or, alternatively, that we just wanted to test to a very small group for now to use the results later.

That said—this looks very, very cool and I’m excited to use it.
Dave Calleja
June 5, 2009 1:34pm

That. Is. Awesome.
Naca-Yoda
June 5, 2009 1:40pm

Very tidy! Well done!
Chris McMahon, Cincinnati
June 5, 2009 1:43pm

Awesome, thanks for giving me another benefit to sell.
David Greiner
June 5, 2009 1:49pm

Thanks guys, happy to hear you’ll find this useful.

@Matthew, thanks for the suggestion. In an effort to keep things simple, you could just set the recipients of your campaign as a smaller segment of the entire list. That way, you’ll only be running the test on (and sending the winner to) a subset of your entire list.

Of course, like many feature we’ve launched it will more than likely change and improve based on continued customer feedback. We’ll be starting simple.
Alex
June 5, 2009 2:29pm

Really nice implementation—I’m looking forward to putting this to the test, as it were.

At risk of looking a gift horse in the mouth, is the option to test delivery time anywhere on your horizon?
David Greiner
June 5, 2009 2:44pm

@Alex, we’ve built things in a way that we can add additional test criteria if the demand is there, so it’s certainly something we’d consider. Thanks for the suggestion.
Finge
June 5, 2009 3:08pm

This will be a very nice addition indeed! Can’t wait. I love your approach and the fact that you are striving to keep it simple and clean!

Keep up the good work!
Simeon Duncan
June 5, 2009 4:17pm

That is awesome. Can’t wait to have a play and have a test. Keep up the great work at Campaign Monitor!
Dimitris Rakopoulos
June 5, 2009 5:52pm

Great, can’t wait!

Next on the To-Do list: Autoresponders + Triggers!
Alex van Ginneken
June 5, 2009 9:12pm

This is great functionality! Can’t wait!
Nippr
June 5, 2009 9:18pm

Great work. When delivery time gets added this is awesome.

Suggestion would be to be able to test more than just two versions. Not that I think 20 would be very cool, I could find a use for three versions.

Looking very much forward to this! Thanks.
Julian Wellings
June 5, 2009 9:52pm

Bring it on! I like the way CM doesn’t just copy what everyone else is doing but looks hard at how to improve on it.
Joshua Clanton
June 6, 2009 12:05am

Looks amazing, guys! I second Matthew’s suggestion that auto-sending the winner be optional, but not required. That way we can tweak further if the results aren’t quite what we were looking for.
Eric Oliver
June 6, 2009 1:00am

LOVE it!
I have only one feature request (and am seconding some of the other people above): Allow time of send to be one of the testing criteria.

Most of my A/B testing in the past has been determining effective times to send to my list.

Thanks again - very exciting!
Jason
June 6, 2009 8:32am

Fantastic, and good timing! I was just contemplating the hassle of doing my own AB testing with the current template system (No duplicate previous template/campaign?)

I can only second/third the above comments RE a manual ‘big’ send rather than fully automatic. Especially as this feature would also be really useful for reporting back to clients who may wish to do further changes, testing etc.
Smallfurrything
June 6, 2009 8:56am

@Jason If you go to create/send there’s an option on the right to create a new campaign based on one you’ve sent. I can’t remember the exact wording.

But this looks like a great addition CM, well done!
steven
June 6, 2009 11:46am

This is a great idea. I cant wait to dive right into it!
Brad Haynes
June 7, 2009 2:37pm

Yeesh, you guys are so smart ;)
Simon
June 8, 2009 6:49am

Fantastic stuff. Can’t wait to try it out.

My addition to the wish list echoes the calls above for delivery time to be one of the criteria. To be able to show clients that a campaign achieves a better open rate at 9am versus 5pm (or not as it may be) would be a great help.
Anna Yeaman
June 8, 2009 11:16am

This was no.1 on my wish list so I couldn’t be happier! Can’t wait to start using this.

Agree with all the above suggestions about being able to test delivery time. This comes up quite often with my clients who seem obsessed with best time to send….
James
June 8, 2009 5:59pm

Hi guys… great news!
Will the Google Analytics tags for the campaign name etc follow through with whether it was test A/B…? Again, useful in tracking conversions right the way through…
Simon
June 8, 2009 10:18pm

Wow, this is excellent! Can’t wait to use it.
Al Stevens
June 9, 2009 12:34am

Oh, just so so marvellous. Do we know when it’ll be ready!
Restoration Media
June 9, 2009 10:55am

Awesome! This is truly awesome. I can’t wait to be able to try this out.
David Greiner
June 9, 2009 11:27am

Thanks for all the requests for the ability to test best time to send, we’ve definitely taken that on board.

@James - Yep, we’ll be appending a version number to the Google Analytics tags so you can track each version separately in your GA account.

@Al, we’re hoping to have this released some time in the next month. We’ll be sharing more here as we get closer.
John Driscoll
June 11, 2009 1:21am

No more guessing. I love it!!
Brandon Eley
June 11, 2009 1:29am

Wow, awesome! Can’t wait to try it out.
Ryan
June 11, 2009 1:37am

Why manually choose the size of the test subset?

Maybe the better way would be to send out until a statistically significant result was obtained (which could be just 10 if one design sucked or could be 10,000 emails if they are close) and then automatically or manually send the winning design to the rest.
Morgan
June 11, 2009 4:34am

Killer feature, and dead simple to use. Best UI ever.

Are you guys thinking of Salesforce integration, dynamic content or drip feed campaigns?
Jon S
June 11, 2009 4:42am

Sweet UI, a welcome addition to the product.
Julia@Niche
June 11, 2009 11:00am

Have I ever told you guys how much I love you? Your consistent innovation ensures that Campaign Monitor is a quality product that works as well for us as it does for our clients.

Also, when will Lunch 2.0 be released to the public? Seriously.
Basti
June 11, 2009 9:24pm

How much will it be? ;-)
Basti
June 11, 2009 10:34pm

Also agree with Matthew regarding the option not to send the winner automatically. If the differences are more complex I can imagine that I would like check them manually and than decide.
Chriz
June 12, 2009 5:07pm

Nice. Looking forward to see this in action
Rob Jones
June 12, 2009 6:32pm

I’ll second @Dimitris Rakopoulos for Autoresponders and most importantly Triggers!

Also, would love to see A/B testing on delivery time as also mentioned above several times…
Elaine Nelson
June 13, 2009 2:05am

I’m so excited about this! A/B testing has always been of a brainbender, and this should really help!
Jon M
June 14, 2009 1:55am

This is REALLY impressive. A genuinely useful, added value feature with simplicity and focus. Look forward to receiving the launch email!
Jeff
June 18, 2009 9:19pm

Absolutely awesome.

Quick Q: What kind of pricing model will be applied to this new feature?
David Greiner
June 19, 2009 9:02am

Jeff, sending an A/B campaign will be just like sending a regular campaign. Even though we’re effectively sending 3 different campaigns, you only pay the usual $5 delivery and 1 cent/recipient.
makonja
June 21, 2009 6:19pm

I’m drooling about this feature. can’t wait to use it. will ease my marketing efforts a lot. Please make it fast :) thanks guys
makonja
June 26, 2009 7:52pm

we are daying for this A/B testing. do you have an approx eta?
Charles Garrison
June 27, 2009 2:27am

Very cool! I’ve been waiting for this feature since we started using Campaign Monitor :)
makonja
July 2, 2009 5:39am

any ETA yet?
Jon Moss
July 4, 2009 9:58pm

How’s the ETA looking chaps?!
Yours excitedly,

Jon :)
Rob Jones
July 6, 2009 5:44pm

Yup I’m still keen to know how that elusive ETA is coming along… Any ideas please?

Rob.
David Greiner
July 6, 2009 5:50pm

Hey guys, testing is coming along nice and we’re on the home stretch. We want to ensure the entire process runs like a dream, and we’re almost there. A release don’t won’t be more than a week or two away, and we’ll make an announcement in the blog as soon as it’s confirmed so you have plenty of notice.

Great to see you all sharing our excitement to get this out in the wild.
Mike Bowzeylo
July 7, 2009 3:17pm

Yay…I can’t wait. This is going to be a stellar feature.
Chris
July 16, 2009 2:37am

I’m glad i saw this as it was one of the things keeping a client off CM. Hope it comes soon!

1 2 >

Comments for this entry are closed.

Browse the Blog

Explore the Email Gallery

@HunterOwens And sorry again! This one has been frustrating for everyone, but obviously for affected customers the most!

Powerful API

Interact and “sync” with your account via simple web-based methods.
Create a free account

Sign up, kick the tires and start sending in less than 60 seconds.
Not just good looks

Track every aspect of your campaigns like opens, clicks, forwards, sales and more.