Blog: Observations & Answers

This is a heads up about an upcoming payment gateway maintenance window starting at 9am on Sunday, May 22nd (CST) / just after midnight on Monday, May 23 Sydney time (see this in your timezone).

During this 6-hour period, some customers and clients will not be able to pay for (and therefore not send) campaigns, or purchase credits. This is a pretty low traffic time but if you have campaigns planned around that time, read on for your options.

Will my campaigns be affected?

If the client sending the campaign is set to pay monthly (and the billing period has started), your campaigns will not be affected. Even if your next monthly payment is due on the 22nd, we'll extend your month into the next day and charge your card then instead.

If the client (or the main account owner) is paying per campaign or purchasing credits then payment will not be able to be taken.

What should I do to avoid being delayed?

You have a couple of options. One way is to just move your campaign send to happen either before or after the maintenance window, so that payment can be taken normally from your card.

Alternatively you (or your client) could pre-purchase enough email credits to send your campaign. Remember, you'd need 1 credit for each person you want to send to, plus 500 credits delivery fee. If there is a chance your list will grow before the send date, purchase some extra credits to be safe.

Only a small number of customers are likely to run into any delays at all, but if you have any questions or concerns, send us an email via support and we can work with you to make sure your email goes out on time.

Since we pushed an update live on Sunday this week (including custom domain support for social sharing and a stack of other new things we'll announce soon), some customers have been seeing some very unusual, rare and very difficult to diagnose errors in their accounts.

This has been resulting in some images not appearing in campaigns, problems importing your designs and the occasional and unhelpful 503 error. Our engineers and system administrators have been hard at work since then trying to track down this gremlin once and for all. Typically, the issue only surfaces when we're under heavy load (the worst time, basically) and while we've managed to cut back the number of errors, the team is still working around the clock (and I do mean around the clock, it's 12.30am in Sydney right now) to knock this on the head.

We've been doing our best to keep everyone in the loop via Twitter and email, and we'll continue to post updates here as we make further progress. There is simply no excuse for this sort of experience, and we want you all to know we're doing our best to resolve it once and for all. More details to follow soon.

March 10, 1pm update: The good news is that we haven't experienced any additional reliability issues with the app since last night's update. In the mean time, we've ordered a pile of new hardware to better distribute the load as we keep narrowing down the issue. Our engineers are pouring through the code, log files and lots of other monitoring reports we have in place as we get closer to isolating this once and for all. More updates will follow once I have something new to report.

March 10, 8pm update: We've continued to avoid any issues today, as a number of safeguards we have in place are automatically spreading the load before a problem eventuates. Hardware upgrades are close and will have a big impact when we bring them online. We've also narrowed down the issue to specific part of the app, which the entire team is focused on right now.

March 11, 5pm final update with some great news: After days of intense testing, log file digging and hard work we've now isolated the issue that's been causing all these headaches. This was a sneaky combination of hard to detect faulty hardware and some resource intensive code changes we pushed on the weekend. We've taken this hardware out of our infrastructure, added significant amounts of new hardware and pushed a pile of optimizations to existing code to make sure this problem can't surface again.

I want to thank everyone for their patience as we hunted down this problem. It was an extremely difficult challenge, and I'm really proud of our team for the long hours and brainpower they put into resolving it. Of course, we're not stopping here. This issue has given us a stack of ideas on further improvements we can make to our architecture so if a problem like this ever does eventuate again, it will be easier to isolate and won't have an impact across the application. This process is already underway. Have a great weekend everybody.

UPDATE: The block has been removed, thanks to MessageLabs for sorting this out with us. We now know the campaign that was the cause and are following up with that customer. Emails will no longer be bounced by MessageLabs because of this block (though normal content filtering still may apply).

Over the last few days we identified an unusual number of bounces for some customers' campaigns, and discovered that some end recipients who were using MessageLab (Symantec's email filtering service) were not getting emails from our servers.

Our deliverability team responded immediately to find out why the bouncing was happening, and to request a delisting. These kinds of listings can sometimes happen if a particular campaign triggers too many complaints, or is flagged by some heuristic filters, but generally they are very short term and easily resolved.

In this case, it is taking much longer than expected to get back to normal service. We have escalated this issue to Director level with Symantec, and they are working with us now to make sure that the original cause is dealt with appropriately, and that all our other email will be able to be delivered normally.

We're sorry if you or your clients have had your campaigns affected by high bounce rates. Please rest assured this is something we take really seriously and we're already working on improvements that would reduce the impact of any similar occurrence in the future.

Who is affected?

If the subscribers on your lists use MessageLabs, and they have not whitelisted our servers in any way, they may currently be bouncing your emails back. In your reports it will show as "Mail Block: General" (though not every instance of a general mail block is due to this issue). MessageLabs seems to be much more used in Australia and NZ, and in financial institutions particularly, so that could be an indicator.

What can you do right now?

If your emails are being bounced by certain domains, and you suspect this is the problem, the immediate solution is to ask the recipient to contact MessageLabs and ask for a whitelisting to get the email they opted-in to. That will fix the problem right away for that specific domain, while we work on sorting out the root issue.

Compensation for bounced emails

If you or your client have been affected, contact us via support with the details of client and campaign, and we will take a look and give you sending credit so you can resend those emails once it is clear at no cost.

We're working on getting this fixed up as soon as we possibly can, and we appreciate your patience. We'll update this post once we have more information for you.

There have been a couple of free-range conversations lately as to whether connecting via Facebook (and other social networks) equals permission to send email campaigns to your friends, likes and logins via an email service like Campaign Monitor. It doesn't help that Facebook Login in particular allows you to export the personal information of your connections, or uses rather ambiguous wording regarding permission on their screens:

Now, we wanted to get one thing clear:

When a person opts-in to receive email from a company or contact via Facebook Login, this constitutes permission to send and receive email exclusively from the Facebook application.

Same goes for social networking sites like LinkedIn, where folks explicitly opt-in to receive updates from contacts via the site itself, but not necessarily from a 3rd-party mailing list.

Note that this is different from adding an email subscribe form to your Facebook 'likes' Page, which is a fine way to encourage folks to sign up for your newsletter.

Why so serious?

From the point of view of a visitor to a site with Facebook Login, there's a good chance that they want to login and receive personalized content, but don't necessarily want to join a 3rd-party mailing list. We've taken this into account in our permission guidelines, which require that all subscribers have explicitly opt-in to receive email campaigns, prior to being added to a list. For more info, here are some answers to common questions about permission.

How about if I want to use Facebook Login on my site?

There are lots of sites now that use Facebook Login instead of rolling their own authentication or membership system. We're totally cool with this, as long as Facebook login is not used as a substitute for an email subscribe form.

If you don't already have a subscribe form on your site, add one. You can then message your existing friends, likes and logins within Facebook and encourage them to sign up to receive your email newsletter via the form. It's easy and you can know with certainty that you're on the right side of the tracks when it comes to permission.

Finally, if you have any questions about who you can and can't email, check out our anti-spam policy or get in touch with the good folks in support - we're always happy to help with those tricky permission questions and provide solid pointers along the way.

Despite the odd request for an opinion, we've been reserved when it has come to speculating about what Facebook Messages will mean to email designers and their clients. Facebook announced this under the ominous 'Project Titan' label in mid-November - predictably, over two months (and much hysteria) later, we're only just starting to see these updates get rolled out, let alone the 'emailpocalypse'.

Finally, a couple of folks on our team gained access to Facebook Messages and to be honest, the sentiment has been pretty flat. Our friends at Litmus obviously felt the same when they got a early sneak-peek and declared it to be 'irrelevant to email marketers'. But the awkwardness of not having subject lines in replies and segregation to the 'Other' pile aside, we thought we'd provide some practical observations on how Facebook Messages deals with email campaigns.

Text yourself before you wreck yourself

The first thing you'll notice when looking at an HTML email in Messages' 'Other' inbox is that the plain-text alternative is displayed by default. The column width is roughly 60 characters wide, meaning that Facebook forces lines of text to wrap if they extend beyond this:

It's often really easy to ignore the text version of your campaign (unless you're sending plain-text emails, of course), so this is a solid reminder that it's worth putting a bit of effort in, also for the sake of mobile device users and anyone who prefers to read their email sans HTML support. We've even got some plain-text templates and formatting tips to help you back on the path.

Note that Messages does not automatically turn URLs into links, which to us is a real usability boo.

Not quite the CSS support we were hoping for

There's a little 'expand' link next to the plain text version, that loads the HTML content of an email campaign in a lightbox window. The link may not be the most obvious thing in the world, but after the initial coverage on Facebook Messages, I guess we were glad to see HTML email support there at all. Or, so we thought.

You see, every month, we send out an HTML email newsletter. We test it. We keep it simple. Here's what it looks like in most email clients:

With Facebook overriding, or ignoring our CSS styles, we were in for a bit of a surprise:

Admittedly, our newsletter isn't perfect and we haven't tested Facebook's CSS support down to the smallest attribute. But as we all know, it only takes a few properties to knock down all the fences. So far, we've found that:

• padding is not supported, thus the collapsed sections
• background-color is also not supported
• Link colors are overridden with color #3B5998
• p tags have a margin of 10-15px applied to both the top and bottom (unless explicitly defined in your styles)
• Some class names are stripped, so please inline your CSS

I'm sure there's more to find, but those were the most visible rendering issues. Overall, we'd say Messages' HTML email rendering capability is on par with Gmail's - not bad, not awesome - with a few points in the positive for having images turned on by default, but a few points docked for fickle CSS support.

Should I be concerned about Facebook Messages?

I personally don't think this version of Facebook Messages will become a rising star in the email client world. Geared towards short form, text-only messages, its certainly a step up if you value chattering within your Facebook friend network (and its text messaging support is a bonus in this regard), but its limitations as an email platform will most likely prevent it from being widely used as an everyday email address. Plus, would you take a job applicant really seriously if they were sending from a @facebook.com address? Really?

Then again, Facebook Messages may be the unified communications solution of the future and I may live to eat my headband. But for a couple of reasons, I don't think that will be the case anytime soon.