Browse by...
Home Resources Blog

Today at 10am London time (2am in San Francisco and 9pm in Sydney) the Campaign Monitor network was unavailable to all customers for approximately 50 minutes. Everything is back up and running smoothly now, but we wanted to give everyone an honest explanation of exactly what happened and what we’re doing about it.

Here’s what happened

First of all, any downtime is inexcusable. We’re incredibly sorry for delaying that important campaign of yours or temporarily breaking a link in an email you recently sent.

We have multiple load balancers sitting in front of infrastructure at both our data centres that acts like a cop, directing traffic to the right places internally. We renewed our license key with the vendor that supplies this special hardware recently, and it turns out they made a mistake and gave us a key that expired much earlier than they should have. Because the same key is used at both data centres, this effectively crippled our platform the moment they stopped working.

This means that your account itself was unavailable, along with our tracking system and API for that time period. Which is inexcusable.

Here’s what we did

The moment the app went down our operations team were on to it and instantly contacted the vendor. While we would have liked things to move faster at their end, it took some 20 minutes to generate and provide the new keys. The moment we had them in our hands our records were updated and the app was instantly available again.

Here’s what we’re going to do

It’s extremely frustrating that this problem was ultimately due to a trivial error by a vendor we rely on, but we’re not going to pass the buck. This one is on us. Our next steps will be to audit all our licensing across all devices that pose this kind of risk in both data centres to ensure this can never happen again.

I also wanted to say a huge thanks for everyone’s understanding during the 50 mins or so we were unavailable. While the team was working feverishly to get everything back to normal, I can tell you that it’s a big help when you have people letting you know they understand and passing on best wishes. We owe it to you guys to never let a simple thing like this have such an impact again.

This blog provides general information and discussion about email marketing and related subjects. The content provided in this blog ("Content”), should not be construed as and is not intended to constitute financial, legal or tax advice. You should seek the advice of professionals prior to acting upon any information contained in the Content. All Content is provided strictly “as is” and we make no warranty or representation of any kind regarding the Content.
Straight to your inbox

Get the best email and digital marketing content delivered.

Join 250,000 in-the-know marketers and get the latest marketing tips, tactics, and news right in your inbox.

Subscribe

Get started with Campaign Monitor today.

With our powerful yet easy-to-use tools, it's never been easier to make an impact with email marketing.

Try it for free