Home Resources Blog

Today at 10am London time (2am in San Francisco and 9pm in Sydney) the Campaign Monitor network was unavailable to all customers for approximately 50 minutes. Everything is back up and running smoothly now, but we wanted to give everyone an honest explanation of exactly what happened and what we’re doing about it.

Here’s what happened

First of all, any downtime is inexcusable. We’re incredibly sorry for delaying that important campaign of yours or temporarily breaking a link in an email you recently sent.

We have multiple load balancers sitting in front of infrastructure at both our data centres that acts like a cop, directing traffic to the right places internally. We renewed our license key with the vendor that supplies this special hardware recently, and it turns out they made a mistake and gave us a key that expired much earlier than they should have. Because the same key is used at both data centres, this effectively crippled our platform the moment they stopped working.

This means that your account itself was unavailable, along with our tracking system and API for that time period. Which is inexcusable.

Here’s what we did

The moment the app went down our operations team were on to it and instantly contacted the vendor. While we would have liked things to move faster at their end, it took some 20 minutes to generate and provide the new keys. The moment we had them in our hands our records were updated and the app was instantly available again.

Here’s what we’re going to do

It’s extremely frustrating that this problem was ultimately due to a trivial error by a vendor we rely on, but we’re not going to pass the buck. This one is on us. Our next steps will be to audit all our licensing across all devices that pose this kind of risk in both data centres to ensure this can never happen again.

I also wanted to say a huge thanks for everyone’s understanding during the 50 mins or so we were unavailable. While the team was working feverishly to get everything back to normal, I can tell you that it’s a big help when you have people letting you know they understand and passing on best wishes. We owe it to you guys to never let a simple thing like this have such an impact again.

  • Wil Benton

    Thanks for the extremely quick heads up (and fix) guys, appreciate it!

  • René

    Why not use 2 different license keys on each datacenter? :)

  • Katrina Youngman

    Great to see the speedy response time, and super impressed that you guys managed to get a blog post out explaining the situation so quickly. Love your work!

  • Ben Richardson

    @René – our Vendor currently doesn’t allow separate keys but we’ll be discussing this with them.

  • Niki

    Thanks Campaign Monitor team – appreciated all the tweets and the fast response to the problem.

  • Pascal Wheeler

    Appreciate the speed of resolving the issue and explanation, thanks David.

  • Ola Nate

    Good job still. Speaking from experience 20 mins is good. Thx

  • Jamie

    And honesty like this is exactly why we love you guys!

  • Juan

    Appreciate the speed of resolving the issue and explanation.

    Still… you state in the website that there’s redundancy in every layer of the system. If the load balance system fails for other reason than expired key (say hardware issues), then the service goes down as well (or at least reduced to half it’s capacity which mean lots of timeouts and delays).

    I’m not trying to be negative here but… what if the vendor couldn’t give you the new key for 12 hours? You should have had a way to solve it without the new license key.

    As I said, the transparency and communication is appreciated but I thought the constructive critique might help in the future

  • John

    To quote Big Boi; “I got a back up plan to the back up plan to back up my back up plan.”

  • Yakov

    It happens, don’t worry about it. We love you guys. CampaignMonitor 4 life.

    -Yakov @ http://MixedInKey.com

  • Ben Richardson

    @Juan – We do have redundancy at the load balancer level with excess capacity. In fact, we would have been fine if one of the load balancers burst into flames!

    But we didn’t account for the license key issue which can take out all levels of redundancy instantly. We will be improving on this.

    We completely agree that constructive criticism is required and we really appreciate your feedback.

  • Jessica

    Still having trouble this morning, or is it me???

  • Stig Morten Myre

    @Jessica, if you’re still having trouble, could you email us so we can investigate?

  • Nikola Kumburovic

    Jessica is not the only one.

    Sign Up process on Holler.ie and other sites are still down.

  • Kris

    I can login still. Login link on site and our custom url are down. Support is insisting its fine but clearly its not…maybe dns needs to update because of outage last night? I doubt it but not sure what the issue is. Why can they see my login page but not me? I’ve tried multiple browsers. Only thing that works is my iphone.

  • Kris

    Sorry…meant “CAN’T” login still.

  • Diana Potter

    @Kris, That is odd! I believe we’re talking to you right now via support. Hopefully we can get to the bottom of this for you. The fact that it’s working on your iPhone but not elsewhere is pretty odd but means it could be something on your network blocking the page.

    @Nikola, the issue with holler.ie actually looks like a problem with that CNAME record. Whoever owns that domain will want to fix the DNS records for that and restore the CNAME record that points to our servers.

    If anyone else is still having issues please do get in touch with support. There shouldn’t be outstanding issues.

  • Kris

    Its working over wifi…but not our cat5…sorry…I guess an issue on this end…no clue how thats possible.

This blog provides general information and discussion about email marketing and related subjects. The content provided in this blog ("Content”), should not be construed as and is not intended to constitute financial, legal or tax advice. You should seek the advice of professionals prior to acting upon any information contained in the Content. All Content is provided strictly “as is” and we make no warranty or representation of any kind regarding the Content.
Straight to your inbox

Get the best email and digital marketing content delivered.

Join 250,000 in-the-know marketers and get the latest marketing tips, tactics, and news right in your inbox.


See why 200,000 companies worldwide love Campaign Monitor.

From Australia to Zimbabwe, and everywhere in between, companies count on Campaign Monitor for email campaigns that boost the bottom line.

Get started for free