Queen Bee simplifies billing tasks for 37signals customers

From Signal vs. Noise:

Queen Bee is the name of our new, internal, unified billing, admin, and stats platform. Here's an example of how it will make things easier for customers:

So, let's say you have two Basecamp accounts and a Highrise account and you want to update your credit card because it's about to expire. Before Queen Bee you'd have to log into each account separately, click the Account tab, click the "change card" tab, and update your card. But now all you have to do is pick any account to update and you'll see a screen that look like this:

You'll see that if you have multiple accounts with us on the same credit card you'll be asked if you want to update all the accounts or just one (or two) of the accounts. This makes our customer's administrative tasks (updating cards, for example) a lot easier. Hassle be gone.

Read the whole post to learn more.

Where to get status updates from 37signals

If you're ever having trouble reaching a 37signals site (including our products), then check the 37signals System Status page. We'll always post the latest news there on any service issues. The 37status Twitter stream will also keep you up to date on the latest news. Since both of these are hosted independently, we can post updates there even if there's an episode with our normal servers.

What happened this morning?

All the 37signals properties were offline for two hours this morning between 10AM and 12PM CST (16:00 to 18:00 GMT) as our load balancer blew out and knocked out the network connection for all our servers. No data was lost and the machines all kept running, but they weren't accessible from the internet.

We're very, very sorry for this interruption of service. While we were able to report on the progress of this interruption through our http://status.37signals.com (all the products and 37signals.com pointed to that site during the majority of the outage) that's a small consolation when you want to access data stored on our services right now. It was just not good enough.

While we don't have a formal service-level agreement (SLA), we still want to compensate anyone who felt they were negatively affected in their work because of this outage. Please write support@37signals.com and we'll get that taken care of.

Naturally, we're going to have a long, serious talk with our service provider (Rackspace). They're supposed to be the best in the business, but in this instance they failed us, so we in turn failed you. We'll do everything we can to make sure that something as simple as a load balancer (or firewall or switch or any other network equipment) going bad does not cause two hours of downtime.

Again, we're truly sorry for this interruption. This is not how Fridays are supposed to be.

New 37signals Twitter stream

We're experimenting with a Twitter stream: Twitter.com/37signals. We plan on posting status updates for 37signals products, occasional links to interesting posts, and, well, we'll see. If you have a suggestion for what you'd like to see there, let us know.

Are you also reading Signal vs. Noise?

You already know this Product Blog is the place to turn for news on 37signals products. But did you also know that Signal vs. Noise, our company blog, serves up a daily dose of our thoughts on design, business, experience, simplicity, the web, culture, and lots more? It was recently ranked #1 at a list of the top 100 user-centered blogs. Grab the RSS feed and see why over 78,000 people have already subscribed.

Downtime explanation

At around 7:40pm CST November 12, our products, forums, and email support systems went offline. The downtime lasted approximately 2.5 hours. The good news: No data was lost.

We apologize for the downtime

We're deeply sorry for the downtime. We understand you rely on us to be reliable. We apologize for letting you down on this occasion. We will be reevaluating our disaster preparedness in response to this situation.

Here's what happened

At about 6pm CST, Rackspace, our primary data center provider, experienced a significant "power event" which ultimately included a dual cooling system failure and multiple transformer shutdown.

Here's the chain of events as reported by Rackspace:

1. At approximately 6pm CST a traffic accident caused damage to a power transformer which provides utility power to the Rackspace data center in Dallas, TX.

2. Power automatically switched over to backup generators without disrupting service for any customers.

3. When generator power was established two chillers within the data center failed to start back up.

4. Utility power was re-established through a secondary utility source.

5. As a result of temporary data center temperature increases, Rackspace proactively shutdown a number of customer servers to protect them from overheating around 7:40pm CST.

6. As of 8:45pm CST the chillers were back up and running and we are operating on generator power throughout the data center. Servers are being powered back up as the temperature drops.

Our servers began being restored at approximately 10:15pm CST.

Thanks for hanging in there

Will be using this situation as both a wake-up call and a learning experience. While our systems are engineered to chug through major failure, this "perfect storm" chain of events beat both our set-up and our data center's sophisticated back-up systems. We will work hard to further diversify our systems in order to make any future downtime event like this even more rare.

We apologize for any inconvenience this downtime caused your business. If you feel you were significantly impacted by this downtime, please send an email to support and we'll credit you for the downtime.

Put it in Basecamp b*tches!

Spotted at Flickr:

Caption: "from claribelle's cube, a PM after my own heart. it don't exist if it's not in the queue."

Related: Basecamp Screengrab Confab Pool [Flickr]

Amazon Web Services "success story" on 37signals

Amazon Web Services recently published a success story about 37signals' usage of Amazon Simple Storage Service (Amazon S3). 

From a technical perspective, [37signals' David Heinemeier] Hansson states, "S3 makes hosting massive amounts of files easy, affordable, and a no-brainer. To turn up our storage offerings no longer involve planning how to extend our cluster or deal with huge backups. It's simply a pay-as-you-go proposition. Couldn't have been easier." From a business perspective, Founder Jason Fried says, "I thought S3 was brilliant, I like low risk, outsourced services that give us room to grow with very little initial outlay."

Amazon S3 ensures the uploaded files of 37signals' customers are reliably stored in Amazon.com's robust storage center and provides the ability to scale to an unlimited capacity. S3 has been a great, seamless solution to our hosting needs and helps us keep costs down and price our products affordably.