Are you A/B testing your email subject lines?

If so, you are taking a step in the right direction towards better results from your email marketing.

However, if you are just creating two different subject lines at random for each campaign then you may not be getting the best results.

So when CoSchedule, a startup that uses Campaign Monitor to send their popular email newsletter The Content Marketing Update, recently wrote about how they systematically A/B test email subject lines to find out what works for their audience – we knew it was something we had to share with you.

In this post, we outline the step by step process CoSchedule use to test subject lines with the hope that you can steal their process and improve your email marketing results as well.

A step by step process for testing subject lines

Step 1: Decide what you want to learn

The first step towards systematically testing subject lines is deciding what you want to learn about your audience.

To do this, try to think of all the questions or uncertainties you’ve ever had when writing subject lines for email campaigns:

  • Do long or short subject lines work best for my audience?
  • Does including a number in the subject line get my audience to open emails?
  • Does subject line personalisation work for my audience?
  • Should I phrase the subject line as a question rather than a statement?

Once you’ve worked out all the things you want to learn, you can then turn that into a spreadsheet for measuring the results.

For instance, if you wanted to know whether long or short subject lines work best for your audience then you’ll need to add a column for ‘Number of characters’ to record the effect this has on key metrics like open rates.

Let’s look at the spreadsheet CoSchedule used to get a better idea of how this works.

Date Title Opens Unsubs EMV Chrctr Shares
3 17 Apps To Help You Make Ebooks 86 1 29% 31 222
How We Grew Our Email List By 2,000+ 73 3 50% 36 76
10 A No-Fail Method For Writing Blog Posts 126 3 0% 39 633
50+ Places To Repurpose Your Content 99 1 33% 36 278
17 The Total Guide To Sharing Content On Social Media 110 1 44% 50 153
3 Reasons To Use Exclusive Content 108 2 83% 34 145
24 Using Game Theory As A Content Marketing Tactic 108 1 63% 47 156
How Pixar’s Creative Process Can Help Your Content Marketing 97 0 44% 60 2219*

Although what you are interested in may vary, CoSchedule wanted to be able to answer the following questions about their audience:

  • Do higher open rates lead to more social shares of their blog posts?
  • Does having a number in the subject line increase opens?
  • Do shorter subject lines work better?
  • Do subject lines with higher Emotional Marketing Value scores work better?
  • Does asking a question in the subject line increase opens?

Because that is what they wanted to learn, that’s what they decided to measure. You can see they added columns to record number of characters, EMV score and social shares to name just a few.

Step 2: Lay out the testing ground rules

Conducting a single A/B test on two subject lines is easy, both tests go out at the same time, to the same amount of people and the winner is automatically selected for you.

However, when you start trying to compare multiple A/B subject line tests against each other, you start to introduce a lot of variables that can affect your results.

Things like number of recipients, length of test, day sent and time sent can all impact the results of a specific test and therefore give you misleading information if you are trying to compare two different tests together to get some insight.

So before CoSchedule started running A/B tests on their subject lines, they laid out some ground rules to help reduce these variables, including:

  1. Deciding what percentage of their list would get sent the test emails (10%)
  2. Deciding how long the A/B tests will run for (1 hour)
  3. Deciding what time to the A/B tests will go out (10am CST)

By deciding to send the newsletter out at the same time each week – and run the subject line test for the same length and with the same amount of people – they were able to minimise the variables between each test and get more reliable insights.

Step 3: Run the A/B tests

Each week Co-Schedule would send out their newsletter and A/B test two different subject lines against each other using Campaign Monitor’s built-in A/B testing tool.

Given that each newsletter contained different content, every email had different subject lines that were tested against each other, but there were some subtle similarities.

Sometimes they would test using numbers in the subject line, other times they would test using a question as opposed to a direct statement.

Each time they ran an A/B test they would document the results, and over the period of 2 months they built a good set of data on how each type of subject line was performing which allowed them to then analyse the results and pull some insight.

Step 4: Analyse the results and use the insights to improve your email marketing

Now that they had accumulated the results of a number of different subject line A/B tests, they were able to do some digging into the data to learn a bit more about what works for their audience.

For instance, one of the original insights CoSchedule was looking for was ‘Does asking a question in the subject line increase opens?’

By looking at the open rates for all the subject lines that contained questions, they were able to work out that asking a question in the subject line does not work for their audience and in fact decreases open rates.

They were then able to apply this learning to future subject lines, removing questions and improving their email marketing results.

What CoSchedule learnt about their audience

By recording their results over the period of a few months, CoSchedule was able to ask a number of questions of their data and find out what works best for their audience.

Does having a number in the subject line help?

In their testing, they pitted subject lines with numbers in them against subject lines without numbers a total of 5 times. Of those five, three won.

Their conclusion was that numbered headlines did perform slightly better, but not significant enough to say that they are a sure thing.

Do longer or shorter subject lines do better?

In their testing, they found that shorter subject lines were less successful for them than longer ones, and that around between 40 and 50 characters was their sweet spot.

Does asking a question in the subject line get more opens?

Despite what they previously thought, when they tested subject lines with questions against those without they found that the question lost every time.

In conclusion

Just to be clear, the purpose of sharing this with you was not for you to takeaway that the right subject line length is between 40 and 50 characters, or that asking questions reduces open rates.

The purpose of sharing this with you was to show that sometimes popular wisdom and research from different authorities doesn’t always apply to your unique audience, and the only way for you to know what is going to work for you is to systematically test and learn from your results.

So go ahead and follow the steps outlined in this article, as the things you’ll learn about your audience will help you get better results from your email marketing moving forward.

Want to improve your email marketing?

Join over 20,000 other marketers & designers who get tips on improving their email marketing delivered directly to their inbox.

  • Garrett Moon

    Thanks for the great post Aaron! I hope our data was helpful for your readers :)

  • Ros Hodgekiss

    Glad you enjoyed the post, Garrett! We really appreciate you sharing your experiences with us :D

  • Bn Gold

    Great article. I definitely appreciate the step by step approach taken by the author.

    The third and last takeaway of this article (“Does asking a question in the subject line get more opens?”) comes to me as no big surprise. Having worked with a number of marketing automation tools in the past (ex. Marketo, Net-Results, Callidus Cloud Marketing Automation) and having sent many, many automated emails, I have noticed the same results. I attribute this to another aspect to consider when sending emails. It actually precedes the information put forth in the present article :

    • Identify the “From” field as a personalized identity (ex. refrain from using generic Sender info such as “Marketing” []);

    From there, asking a question in your subject line you can achieve some level of success if:

    1) KNOW YOUR RECIPIENTS: your email segments (targeted recipients) are responsive to “electronic conversations”, such as digitally native audiences  (ex. Gen Y’s)

    2) HAVE A RELATIONSHIP: your company has an existing “community” type relationship (if you have a blog for example)

    3) HAVE FUN WITH IT (if your company narrative permits): your subject line does not sound “commercial” to the email recipients. For example, modify “Why should I use Pampers diapers?” to something more relational such as “How can I keep my baby’s bum happy?”.

    Have something to add? PM or Mention me on #Twitter at @GauldBen

    *To CampaignMonitor: thank you for this blog. It is very informative.

  • Aaron Beashel

    Hey Ben

    Firstly, thanks so much for sharing your experience here.

    You’re definitely right that using a question in the subject line definitely works better when you know your recipients and have a relationship with them.

    I’m also really glad you brought up the point of using a personalised From name as opposed to the generic company name. This is something i’ve been meaning to test and see what resonates with out audience so I appreciate the reminder!

    Keep up the good work and thanks again for sharing your thoughts.


  • Windows 7 Activator 2013

    Right here is the right webpage for anyone who really wants to understand this topic. You realize so much its almost tough to argue with you (not that I really will need to…HaHa). You definitely put a new spin on a subject that’s been written about for many years. Great stuff, just great!

Want to improve your email marketing? Subscribe to get tips on improving your email marketing delivered to your inbox.

Join 200,000 companies around the world that use Campaign Monitor to run email marketing campaigns that deliver results for their business.

Get started for free