This evening all apps were offline for about two hours due to a power and cooling failure at the data center in Texas. Hardware was shut down for precautionary reasons so it didn't overheat.
Updates were posted to the 37signals System Status Site. At 9:25pm CST all service was restored.
We are currently awaiting an official incident report from Rackspace, the data center operator. Once we have the official response we will post it here.
We're very sorry for the unexpected downtime. We appreciate your patience, understanding, and continued business.
If you feel your work was negatively affected because of this outage, please write support@37signals.com (and include your application URL) and we’ll handle compensation.
UPDATE: Rackspace just posted the official explanation of the downtime:
At approximately 5:00 P.M. CDT, our DFW data center experienced a brief loss of utility power that required a transfer to generator power. During the transition, two external chillers which run on independent generators, were brought online manually and efforts to switch a third internal chiller to generator power began. The first attempt failed, and a subsequent attempt to bring an additional chiller online was made. At that point, it was decided to transfer the data center over to the second utility feed and bring up an internal chiller on utility power. Once that internal chiller was brought online, there was a failure of an external unit, but the second internal chiller was brought up immediately. During this process the temperatures in the data center quickly increased which caused disruption to some customer equipment.
We were able to identify that the control system for the internal chillers prevented the initial startup under generator power due to a sequencing issue. We have the chiller and electrical contractor’s onsite working with the data center engineering team to resolve the issues that prevented the rapid startup of an internal chiller after utility power loss. We will continue to provide updates to the support teams as the root cause issues are resolved.
Rackspace sincerely apologizes for any inconvenience this incident might have caused you or your customers. Please let us know if you have any further questions or comments around this incident.
