April 29th, 2001


So it appears that the database decided to deadlock. The machine was fine, physically, and the operating system was perfectly fine ... the database software (good ol' MySQL), decided to stop working.

Database systems are supposed to guarantee there are no possible deadlocks. (A is waiting for B, but B is waiting for A... so nothing can ever happen) But ... it didn't. All of the sudden hundreds of queries stopped running, leaving all the webservers backed up, with nothing to spit out.

Total downtime: a couple minutes.

The bad part is that there were error messages all over the site... I think I would've preferred if it was just slow. Also, had I not been around, it could've been much worse.