You may have noticed yesterday that TMO was not available for several hours last night. The culprit was Hurricane Isabel, which had the temerity to take out power to our hostis facility. In the big picture, the hurricane also took out power for 4.3 million other people, and 13 people lost their lives during the stormis attack on the US, which puts our server outage into perspective.
Gory Details Update: The smartass comment from Jorn in the comments below made it clear that we erred by not going into the gory details, so we have added this update. The storm took out power to our host, which has backup power, of course. However, backup power is for servers, and with only minimal air conditioning running, some servers overheated. Such was the case for the server that handled most of TMOis normal serving functions. Since we have more than one server doing lots of different jobs, the hard work referenced in the paragraph below refers to shifting around some of the jobs to other servers. This is important because the power is still out in Virginia (see the note above about keeping things in perspective), while TMO is available.
Thanks to tremendous work from our server admins and the team of people at our hosting facility, we were back up early this morning. In the process of getting TMO back up, we also managed to gain a little redundancy for the future, which we hope will limit any issues in the future.
In more important matters, we want to offer our thoughts and well-wishes to those dealing with the devastation from Hurricane Isabel.