We asked Adam to walk through how Versioning evolved from average to awesome – without losing the minimalist look. So, take it away:
Versioning at v1.0
Recently, we launched something a little bit different: a daily newsletter aimed at offering web developers all the news they need in a simple, attractive and (hopefully) well-written package. It’s called Versioning, and it’s proved to be very popular with our readers!
While we liked the idea of a comprehensive daily newsletter, we weren’t sure how big an audience we would have. So we decided on a modest goal of launching at around 50-100 subscribers, and put the word out via our SitePoint networks. We needn’t have worried – we sent our first email to more than 5000 inboxes.
The email has always been a deliberately stripped-back affair, with no headers, images or buttons to speak of. The idea was to offer as simple an experience as possible – just some content you should read with nothing extraneous.
It looked like this:
But nearly 40 emails into the newsletter’s life, the design was starting to look a little plain, and we decided a re-design might be in order.
The thing is, email design is hard! People read their email in so many different ways, and you have to take into account the constraints and limitations imposed by myriad combinations of browsers, clients, devices and apps. It’s a huge undertaking, particularly if you’re not sure what you want. While we were pretty sure people liked the stripped-back nature of the design, we hadn’t actually done any research to back that up.
Fortunately for us, about the time we started to look at moving to a new design, Campaign Monitor launched a new tool that looked like it could make our re-design planning much easier.
Canvas is a new way to create email designs, with a visual editor including content modules and a drag and drop interface. The templates you can choose from are all responsive and retina-ready, with some gorgeous typography.
Canvas pretty quickly took a lot of the pain out of the process, so much so that it became possible to create several different designs for a given day.
One of the best features was the rich previews the editor enabled — once I’d finished with a given design, the preview pane showed me what the email would look like on the desktop and on a mobile device, all at once.
Always be A/B testing
We’ve always A/B tested with Versioning, usually pitting two different subject lines against each other to try to maximize the number of email opens we receive. There’s pretty much no reason not to, and it’s also an excuse for us to write multiple pun-based headlines, which is always welcome.
So we decided to approach the re-design process the same way: we’d pit new designs built using Canvas against our original design to see which received the most clicks. If a given design did well, we’d then test it against others until we had a good idea what our audience liked.
We also asked our readers to get back to us with their opinion on the new designs. It turns out, they did not need much encouragement! Our readers have strong (and conflicting) font preferences.
A few details on how we ran our experiments: we ran our design tests for an hour, used 50% of our audience as our test subjects, and picked as a winner the template which received the most unique clicks (when we’re testing headlines, we use unique opens). The new template had a short message saying we were tinkering with our design and to get back to us with feedback, but the old design said nothing.
Day 1: Making a Fresh start
We started with the “Fresh” template, which I chose for its simple, clean design and attractive sans-serif font.
On Day 1, the new template easily beat our old one, receiving far more clicks (121 compared with 104 for the old design; a 16% improvement). This result was particularly interesting when paired with the mainly negative feedback we received! Some readers questioned the template’s narrow design, others were not fans of the font or the link highlighting, but there was no shortage of suggestions for improvements.
Some notable comments:
“Font is too small to read. I’m using Nexus 5.”
“Nice design: I like it. The previous design wasn’t really broken, though… ;-)”
“I think the main text is a little too washed-out in that shade of grey and the links a bit too shouty – that said, I prefer even my HTML email to be as close to plain text as possible, I find design often distracts from content.”
OK, so people wanted something new, but not necessarily what we’ve given them. Message received!
Day 2: Images vs. no images
The next day we tried something different, pitting the new template against our original template, but this time adding pictures to a few of the sections to test our assumption that a picture-free look worked best for this kind of content. Our assumption turned out to be correct!
A second re-design, incorporating images, was not as popular as a text-only design.
Our old template narrowly beat the new one (118 vs 114 clicks; a -3% change) and plenty of people complained about the images. To be more precise, they complained about the way the images caused the text to shrink, which was fair enough, and quite a literal example of what we were trying to avoid.
Day 3: Switching templates
OK, back to the drawing board. We decided to try a new Canvas template, settling on “Broadsheet” – a light design with plenty of whitespace – again, going for a stripped back version with no images. This time, the new template won (with 126 clicks, compared with 120 clicks; a 5% improvement), and the comments from readers were far more positive.
Day 4: Another round of image tests
With such a positive result in Day 3, we decided to take the new template and run with it, building two versions: one with a header logo and a small hero image, and one without ornamentation. The image-less version won by a margin of 149 clicks to 132 (or 13% improvement), once again suggesting our subscribers value the content over images.
I noticed how much pride I had in opening the emailAnother thing: Once I finished creating the email, I noticed how much pride I had in opening up the email. It looked modern, fresh, and attractive. While data and feedback are important, there’s certainly a place in the equation for personal pride in sending something you think looks great.
“My opinion on the latest email design – I don’t care for the header images, but i could live with the styled text. I prefer the simple text design that you started out with…”
“I almost like this new design, it’s bigger and adapted well to my layout.”
Day 5: Is it accurate? Test again
OK, so we were pretty sure we’d found the template we were looking for. The data and the responses, quibbles aside, seemed to favor a Broadsheet design free of ornamentation. But we decided to double down on the previous day’s test and make sure we had our winner locked in. We ran the same test and got the same result: the stripped back version was preferred, by a wider margin than the day before (197 clicks vs 173; a 14% improvement).
“I have to say, this looked amazing on my iPhone. Great newsletter – I’m hella getting into it :D”
“The new design of… Versioning is good. It is clearly better than before.”
Day 6: Header and hero image?
While we liked the stripped back design, we decided to make one more attempt at an image-filled design. This time we had two Broadsheet templates, one with a header and a hero image, and one with a header alone.
As expected, the version without a hero image won handily. Interestingly enough, I did not add a message asking for feedback, and none came. Given the ferocity of the earlier complaints, I took this as a win!
A winner is declared
So, our emails have gone from this:
Much nicer, much more modern – and responsive to boot!
Canvas allowed us to quickly build modern, responsive emails, then test which ones our users preferred – something that would’ve taken many hours and produced poorer results, had we tried to roll our own.
Now we’ve settled on an optimized template, our team can get back to finding the best content to put in the newsletter – as well as the best pun-filled headlines, of course.
If you’re interested in receiving such a good-looking email (or just like web design and dev news), why not subscribe?
Want to improve your email marketing?
Join over 20,000 other marketers & designers who get tips on improving their email marketing delivered directly to their inbox.