At around 7:40pm CST November 12, our products, forums, and email support systems went offline. The downtime lasted approximately 2.5 hours. The good news: No data was lost.
We apologize for the downtime
We're deeply sorry for the downtime. We understand you rely on us to be reliable. We apologize for letting you down on this occasion. We will be reevaluating our disaster preparedness in response to this situation.
Here's what happened
At about 6pm CST, Rackspace, our primary data center provider, experienced a significant "power event" which ultimately included a dual cooling system failure and multiple transformer shutdown.
Here's the chain of events as reported by Rackspace:
1. At approximately 6pm CST a traffic accident caused damage to a power transformer which provides utility power to the Rackspace data center in Dallas, TX.
2. Power automatically switched over to backup generators without disrupting service for any customers.
3. When generator power was established two chillers within the data center failed to start back up.
4. Utility power was re-established through a secondary utility source.
5. As a result of temporary data center temperature increases, Rackspace proactively shutdown a number of customer servers to protect them from overheating around 7:40pm CST.
6. As of 8:45pm CST the chillers were back up and running and we are operating on generator power throughout the data center. Servers are being powered back up as the temperature drops.
Our servers began being restored at approximately 10:15pm CST.
Thanks for hanging in there
Will be using this situation as both a wake-up call and a learning experience. While our systems are engineered to chug through major failure, this "perfect storm" chain of events beat both our set-up and our data center's sophisticated back-up systems. We will work hard to further diversify our systems in order to make any future downtime event like this even more rare.
We apologize for any inconvenience this downtime caused your business. If you feel you were significantly impacted by this downtime, please send an email to support and we'll credit you for the downtime.
