How many contacts do you need before it's worthwhile doing a split test? 100s? 1000s?
Hey Kris, it's hard to say what is to be considered an 'significant sample size'. However, given that the A/B tests send the best-performing version to the majority of email addresses on your list, you're likely to get some benefit in running one on say, two competitive subject lines, regardless of list size :)
I'd love to hear other folks' thoughts on this, too - at what size is a list worth A/B testing on?
The confidence that you can have from any sample data is a pretty well-established area of statistics, but not one i would want to spend time (mis)interpreting for you :)
The short answer is that the bigger your list, the smaller the percentage of subscribers you'll need to sample in order to have confidence in the results.
Attaching hard numbers to that is difficult. Unless you sample everyone (100%) your confidence level can never be 100%, so it becomes a game of acceptable error margins. 37signals have a lengthy discussion that is worth reading.
Thanks Ros and Phil. Phil, I'm going to start larger sample sizes on smaller list. So simple, why didn't I think of that? Checking out your other links now.
Love 37 signals :)