365 Main and the July 24th San Francisco Power Incident

Posted on July 25, 2007

Yesterday's power outage in San Francisco that brought down some popular websites is still topping Techmeme. The power outage took down 365 Main, a data center providing web hosting to some major websites. For some reason some of 365 Main's crucial backup generators also failed resulting in outages for Technorati, Craigslist, Six Apart, Red Envelope, Yelp and Netflix. Laughing Squid captured some of the outage messages from these web companies. Valleywag oddly blamed a drunk employee for 365 Main's backup failures but B.L. Ochman notes that posts by Robert Scoble and others quickly refuted Valleywag's strange post. Maybe it was Valleywag's post that was created by an intoxicated employee?

Computerworld says no one is sure exactly what happened to 365 Main's backup.

No one seems to know what went wrong. Maybe 365 Main's backup generators came up but didn't stay up. Maybe there were other problems because power didn't just drop off, but fluctuated. No doubt a lot of questions are being asked.

But it's yet another wakeup call for businesses that rely on the Internet (and that's getting to be just about all of them): Co-hosting isn't perfect. Backups need backups. Contingency plans need contingency plans.

And, most of all, trust has no place in the technology business.

Here are some explanatory posts from some of the websites that were down because of the power outage and/or the 365 Main data center outage: Second Life, Technorati and Six Apart. Craigslist's message included lolcats grammar: "OMG WHERZ MAH POWERZ."

While this outage was not at all heat related there was a serious heat wave in California last year that took MySpace offline. With global warming creating more intense and longer lasting heat waves these power outage problems will escalate. Web companies and data centers need to find ways to combat this potential problem. Making sure backup generators are working is obviously essential.

A post (no longer available) on Data Center Knowledge blames the 365 Main problems on generator failures. 365 Main has also posted a status update about the July 24th San Francisco Power Incident. The update says 365 Main is "still investigating the root cause of why certain generators did not start when called on for back-up power."