Feature request: A/B testing

This is something I think would be very useful when tweaking and analyzing campaigns:

When sending a campaign, one would be able to create two (or more) versions of the campaign, and change some details for each of them, like the subject line, call to action, graphics, etc.

The subscribers one would select for the campaign, would then randomly be divided into equal lists, and each version would be sent to one.

The report would then show statistics for the campaign as a whole, but also compare statistics for the different versions.

That way you can check which subject line gets the best open rate, which call to action gets the biggest click through (and ROI, with a Google Analytics integration :), and so on.

It is of course possible to do something like this already, but it takes a lot of work, and doesn't give as good a presentation of the compared statistics as you could have with this feature.

llong, 10 years ago

I second the idea

Mathew Mathew, 10 years ago

Thanks guys,

Split testing as you describe is something we do get a few requests for, and I'll make sure your votes are added. We can see how it would be useful, so it is something we might consider down the track.

Get in touch with us on Twitter: http://twitter.com/campaignmonitor
We're also on Facebook: http://facebook.com/campaignmonitor
epersonae, 10 years ago

Definitely in favor of this.  Manual A/B testing is driving me bonkers!

liquidmagic, 10 years ago

A/B testing was just introduced on Mailchimp's service. They implement it in an interesting way. You might (I am referring to the CM team) wanna see how it works. Still, they way it is implemented is patented so you can't just duplicate it.

In any case, I vote for this feature as well. Mailchimp support this only for campaigns created starting from their templates and not from custom user HTML templates. That's a pitty!

Dave Dave, 10 years ago

Thanks guys, we've got a few of our own ideas on this. Stig... any hints of sarcasm there? If not, what would you do differently. The more feedback the better guys, if this happens we want align it with your requests as much as possible.

Stig Stig, 10 years ago

No sarcasm at all, Dave :)

I just honestly think their A/B feature is too limited.

In addition to liquidmagic's point about it only working on their default templates, it's also limited to from names, subject lines and delivery times.

These factors are all relevant, but they mainly affect open rates - I would also like to be able to test the contents.

They do get some things right, though:
- they actually have an A/B test feature :)
- it seems really easy to use
- they only allow testing one criteria pr test
- excellent use of an animated gif in their last email:

Anyways, I'm glad you're looking into this.

Make sure you get one of the key parts of this right: Allow split testing between two different HTML-files/plain texts.

And then maybe it would be a good idea to show a little warning if the link urls are different for the two versions?


The Campaign Monitor Blog – HTML email smarts to go with your good looks.
Dave Dave, 10 years ago

Much appreciated Stig, thanks for the quick reply.

tbianco tbianco, 10 years ago

Without a doubt I second, third, forth, and fifth this feature request. Having the ability to send alternating emails to a list is so helpful because it gives you the ability to test what subject or wording produced a better result. As a marketing company this is absolutely needed because I want to know if I say "x" will it get better results for conversions than "y".

Having the ability to take a general email and then split it with different headlines, content, the way you address someone, or having a different layout or graphics would be a great help.

Here's an example of one of our client's websites we are splitting for Search Engine Marketing to test which will capture people's attention when searching for Spinal Decompression Therapy:
http://kechiro.com/ and the other site is http://kechiro.com/b.

In this example we are wanting to see which wording and design people will relate too most. Based on this test it will begin directing how we design and talk about Spinal Decompression Therapy to our market.

Tony Bianco
Veloce Media
Media & Marketing Solutions for Developing Businesses
brighterplanet, 10 years ago

Definitely would love to have this! Freshview guys, what's the status?


Dave Dave, 10 years ago

Hey Andy, we agree it's certainly a useful feature and one we're strongly considering. We'll post more here or on the blog if/when we make a decision on its implementation.

reallysimple, 9 years ago

Yah, this is easily my #1 feature request. It is SOOOO time consuming split testing with the current system, opening that 5k long csv in excel, cutting it into 4th, making 4 campaigns...just way too much time. For split testing to be effective it needs to be easy to do - easy to change variables. The more we grow and depend upon our email marketing, the more we need to split test. If campaign monitor can't deliver this feature soon, we're going to regrettably have to look for another vendor for this service. You guys are awesome - I know you can make this happen.

reallysimple, 9 years ago

I just noticed that this thread is over 9 months long - I really can't believe you haven't been able to deliver this feature in that time.

Mathew Mathew, 9 years ago

Hey reallysimple,

It's not an issue of we 'haven't been able to deliver it', it is an issue of prioritising and choosing what to do. There are many hundreds of other requests, planned improvements, feature update, fixes, enhancements and so on that are also in the mix.

You'll notice that Dave mentioned it is a feature we are considering, and that is still the case. I'll add your vote to our list, thanks for your feedback.

Get in touch with us on Twitter: http://twitter.com/campaignmonitor
We're also on Facebook: http://facebook.com/campaignmonitor
kgrace kgrace, 9 years ago

Another vote for A/B testing!

skriver, 8 years ago

Any news on this feature request? It gets my vote too!

sccr410, 8 years ago

Hands down a feature that is a must. As we have been introducing A/B testing using Google tools for our client websites, they see the benefit and some are starting to ask about the email system (CM). You can think your emails are successful, but you would never know how successful you can be unless you are able to try different things. Besides just the obvious in A/B email testing (subject line & content), being able to break up a single email over the course of a week to determine what day of the week produces the best results would be fantastic.

Dave Dave, 8 years ago

Thanks for the follow up, we agree this is a super useful feature and it's something we're currently working hard on supporting. We'll announce plenty more details when we get closer to launching this, and at this stage can't provide an estimate on when that will be.

Thanks for the feedback, especially about testing for the optimal time to send.

fz, 8 years ago

Adding my vote! :-) Even if mailchimp's implementation is "weak" they have it and promote it. Which means potential clients who have seen it seem to latch onto the buzzword as a must-have feature (even if they don't know what it means!)

ronnyew, 8 years ago

Add my vote please.

I generally like to test :
- HTML versus Plain Text, or
- Send time (usually testing "work hours" versus "out of work hours"), or
- Different versions of the Subject line


ifixit, 8 years ago

I vote for this as well.

I'm not as greedy, however. I just want the ability to split my lists out randomly. I'd like to divide my list into 4 or so randomly allocated groups, then I could run my own A/B testing without any fancy implementation on your side.

Of course, a more elegant implementation would be welcome. I'm just pragmatic and would like something sooner rather than later.

nippr, 8 years ago

I second ability to split test. I know it will be great, when you roll it out, but until you have all the nuts and bolts aligned, how about doing something like ifixit suggests: giving an opportunity to make a segment of randomly selected receivers.

I would be happy with something like: "make af segment of x amount (e.g. 1000) randomly selected receivers". Or "split this list into x amount of segments of randomly selected receivers"

It should be doable in a short time frame (say a couple of weeks), and would really add value.


woodscreative, 8 years ago

This sounds like homework that should be done before creating a campaign in my view.

danavan, 8 years ago

Guys - sounds like this has been on the list for quite a while to add.  I have clients who are seeking this and I want to move everything over to CM, but this is one of the features that preventing us from doing so.

Any movement?

Stig Stig, 8 years ago


It's getting pretty close, have a look at the sneak preview :)


The Campaign Monitor Blog – HTML email smarts to go with your good looks.
  1. 1
  2. 2

See why 200,000 companies worldwide love Campaign Monitor.

From Australia to Zimbabwe, and everywhere in between, companies count on Campaign Monitor for email campaigns that boost the bottom line.

Get started for free