Today at 10am London time (2am in San Francisco and 9pm in Sydney) the Campaign Monitor network was unavailable to all customers for approximately 50 minutes. Everything is back up and running smoothly now, but we wanted to give everyone an honest explanation of exactly what happened and what we’re doing about it.
Here’s what happened
First of all, any downtime is inexcusable. We’re incredibly sorry for delaying that important campaign of yours or temporarily breaking a link in an email you recently sent.
We have multiple load balancers sitting in front of infrastructure at both our data centres that acts like a cop, directing traffic to the right places internally. We renewed our license key with the vendor that supplies this special hardware recently, and it turns out they made a mistake and gave us a key that expired much earlier than they should have. Because the same key is used at both data centres, this effectively crippled our platform the moment they stopped working.
This means that your account itself was unavailable, along with our tracking system and API for that time period. Which is inexcusable.
Here’s what we did
The moment the app went down our operations team were on to it and instantly contacted the vendor. While we would have liked things to move faster at their end, it took some 20 minutes to generate and provide the new keys. The moment we had them in our hands our records were updated and the app was instantly available again.
Here’s what we’re going to do
It’s extremely frustrating that this problem was ultimately due to a trivial error by a vendor we rely on, but we’re not going to pass the buck. This one is on us. Our next steps will be to audit all our licensing across all devices that pose this kind of risk in both data centres to ensure this can never happen again.
I also wanted to say a huge thanks for everyone’s understanding during the 50 mins or so we were unavailable. While the team was working feverishly to get everything back to normal, I can tell you that it’s a big help when you have people letting you know they understand and passing on best wishes. We owe it to you guys to never let a simple thing like this have such an impact again.