What numbers do you need for split testing?

How many contacts do you need before it's worthwhile doing a split test? 100s? 1000s?

roshodgekiss roshodgekiss, 1 year ago

Hey Kris, it's hard to say what is to be considered an 'significant sample size'. However, given that the A/B tests send the best-performing version to the majority of email addresses on your list, you're likely to get some benefit in running one on say, two competitive subject lines, regardless of list size :)

I'd love to hear other folks' thoughts on this, too - at what size is a list worth A/B testing on?


Get in touch with us on Twitter: http://twitter.com/campaignmonitor
We're also on Facebook: http://facebook.com/campaignmonitor
Phil Phil, 1 year ago

Hi Kris,

The confidence that you can have from any sample data is a pretty well-established area of statistics, but not one i would want to spend time (mis)interpreting for you :)

The short answer is that the bigger your list, the smaller the percentage of subscribers you'll need to sample in order to have confidence in the results.

Attaching hard numbers to that is difficult. Unless you sample everyone (100%) your confidence level can never be 100%, so it becomes a game of acceptable error margins. 37signals have a lengthy discussion that is worth reading.

I've had a brief look at a couple of online calculators. Hopefully the maths is better than the design.

Kris2 Kris2, 1 year ago

Thanks Ros and Phil. Phil, I'm going to start larger sample sizes on smaller list. So simple, why didn't I think of that? Checking out your other links now.

Love 37 signals :)


I have been in love with Campaign Monitor since 2012.

Sign up for free.
Then send campaigns for as little as $9p/m

Create an account