How Sitepoint lifted their email click-through rate by 16% in one day

By Ros Hodgekiss on 11th July 2014

Could you benefit from increasing your email click-through rate by 16% in one day? Adam Roberts at SitePoint recently ran some tests on Versioning - one of their popular email newsletters - and repeatedly improved their click-through rates, while learning about their subscribers in the process. Read on to find out how simple changes can have a big impact on your campaign results.

We asked Adam to walk through how Versioning evolved from average to awesome - without losing the minimalist look. So, take it away:

Versioning at v1.0

At SitePoint, we're a huge believer in email, with a sizeable roster of newsletters containing web design news and views delivered weekly and fortnightly.

Recently, we launched something a little bit different: a daily newsletter aimed at offering web developers all the news they need in a simple, attractive and (hopefully) well-written package. It’s called Versioning, and it’s proved to be very popular with our readers!

While we liked the idea of a comprehensive daily newsletter, we weren't sure how big an audience we would have. So we decided on a modest goal of launching at around 50-100 subscribers, and put the word out via our SitePoint networks. We needn't have worried - we sent our first email to more than 5000 inboxes.

The email has always been a deliberately stripped-back affair, with no headers, images or buttons to speak of. The idea was to offer as simple an experience as possible - just some content you should read with nothing extraneous.

It looked like this:

But nearly 40 emails into the newsletter's life, the design was starting to look a little plain, and we decided a re-design might be in order.

The thing is, email design is hard! People read their email in so many different ways, and you have to take into account the constraints and limitations imposed by myriad combinations of browsers, clients, devices and apps. It's a huge undertaking, particularly if you're not sure what you want. While we were pretty sure people liked the stripped-back nature of the design, we hadn't actually done any research to back that up.

Fortunately for us, about the time we started to look at moving to a new design, Campaign Monitor launched a new tool that looked like it could make our re-design planning much easier.

Canvas is a new way to create email designs, with a visual editor including content modules and a drag and drop interface. The templates you can choose from are all responsive and retina-ready, with some gorgeous typography.

Canvas pretty quickly took a lot of the pain out of the process, so much so that it became possible to create several different designs for a given day.

One of the best features was the rich previews the editor enabled — once I'd finished with a given design, the preview pane showed me what the email would look like on the desktop and on a mobile device, all at once.

Always be A/B testing

We’ve always A/B tested with Versioning, usually pitting two different subject lines against each other to try to maximize the number of email opens we receive. There’s pretty much no reason not to, and it’s also an excuse for us to write multiple pun-based headlines, which is always welcome.

So we decided to approach the re-design process the same way: we'd pit new designs built using Canvas against our original design to see which received the most clicks. If a given design did well, we'd then test it against others until we had a good idea what our audience liked.

We also asked our readers to get back to us with their opinion on the new designs. It turns out, they did not need much encouragement! Our readers have strong (and conflicting) font preferences.

A few details on how we ran our experiments: we ran our design tests for an hour, used 50% of our audience as our test subjects, and picked as a winner the template which received the most unique clicks (when we're testing headlines, we use unique opens). The new template had a short message saying we were tinkering with our design and to get back to us with feedback, but the old design said nothing.

Day 1: Making a Fresh start

We started with the "Fresh" template, which I chose for its simple, clean design and attractive sans-serif font.

On Day 1, the new template easily beat our old one, receiving far more clicks (121 compared with 104 for the old design; a 16% improvement). This result was particularly interesting when paired with the mainly negative feedback we received! Some readers questioned the template's narrow design, others were not fans of the font or the link highlighting, but there was no shortage of suggestions for improvements.

Some notable comments:

"Font is too small to read. I'm using Nexus 5."
"Nice design: I like it. The previous design wasn't really broken, though... ;-)"
"I think the main text is a little too washed-out in that shade of grey and the links a bit too shouty - that said, I prefer even my HTML email to be as close to plain text as possible, I find design often distracts from content."

OK, so people wanted something new, but not necessarily what we've given them. Message received!

Day 2: Images vs. no images

The next day we tried something different, pitting the new template against our original template, but this time adding pictures to a few of the sections to test our assumption that a picture-free look worked best for this kind of content. Our assumption turned out to be correct!

A second re-design, incorporating images, was not as popular as a text-only design.

Our old template narrowly beat the new one (118 vs 114 clicks; a -3% change) and plenty of people complained about the images. To be more precise, they complained about the way the images caused the text to shrink, which was fair enough, and quite a literal example of what we were trying to avoid.

Day 3: Switching templates

OK, back to the drawing board. We decided to try a new Canvas template, settling on "Broadsheet" - a light design with plenty of whitespace - again, going for a stripped back version with no images. This time, the new template won (with 126 clicks, compared with 120 clicks; a 5% improvement), and the comments from readers were far more positive.

Day 4: Another round of image tests

With such a positive result in Day 3, we decided to take the new template and run with it, building two versions: one with a header logo and a small hero image, and one without ornamentation. The image-less version won by a margin of 149 clicks to 132 (or 13% improvement), once again suggesting our subscribers value the content over images.

I noticed how much pride I had in opening the emailAnother thing: Once I finished creating the email, I noticed how much pride I had in opening up the email. It looked modern, fresh, and attractive. While data and feedback are important, there's certainly a place in the equation for personal pride in sending something you think looks great.

Comments included:

"My opinion on the latest email design - I don't care for the header images, but i could live with the styled text. I prefer the simple text design that you started out with..."
"I almost like this new design, it's bigger and adapted well to my layout."

Day 5: Is it accurate? Test again

OK, so we were pretty sure we'd found the template we were looking for. The data and the responses, quibbles aside, seemed to favor a Broadsheet design free of ornamentation. But we decided to double down on the previous day's test and make sure we had our winner locked in. We ran the same test and got the same result: the stripped back version was preferred, by a wider margin than the day before (197 clicks vs 173; a 14% improvement).

Comments included:

"I have to say, this looked amazing on my iPhone. Great newsletter - I'm hella getting into it :D"
"The new design of... Versioning is good. It is clearly better than before."

Day 6: Header and hero image?

While we liked the stripped back design, we decided to make one more attempt at an image-filled design. This time we had two Broadsheet templates, one with a header and a hero image, and one with a header alone.

As expected, the version without a hero image won handily. Interestingly enough, I did not add a message asking for feedback, and none came. Given the ferocity of the earlier complaints, I took this as a win!

A winner is declared

So, our emails have gone from this:

to this:

Much nicer, much more modern - and responsive to boot!

Canvas allowed us to quickly build modern, responsive emails, then test which ones our users preferred - something that would've taken many hours and produced poorer results, had we tried to roll our own.

Now we've settled on an optimized template, our team can get back to finding the best content to put in the newsletter - as well as the best pun-filled headlines, of course.

If you're interested in receiving such a good-looking email (or just like web design and dev news), why not subscribe?

Subscribe to Versioning

Adam is a SitePoint editor, social media manager and lead pun-monger. He's a punny guy, so give him a follow on Twitter and Google+.

Want to improve your email marketing?

Join over 20,000 other marketers & designers who get tips on improving their email marketing delivered directly to their inbox.

2 Comments

  • Designer and geek
    15th July

    Very interesting insights! Thanks for sharing. I’m inspired to try different designs myself.

  • James Marshall
    29th July

    Thank you for this very interesting article. Really enjoyed reading how you used A/B testing as it’s not something we’ve done before.

    I’ve just started using Campaign Monitor so I’m still learning how to analyse the data and improve our open and click rates. This article has given me a boost to try some new things and to learn how to respond to our audience.

    Thank you! :)

Got something to add?

Sign up for free.
Then send campaigns for as little as $9/month

Create an account