June 11th, 2001

Boy did that ever suck

Well, we had planned for downtime tonight for maintenance (at 1:00 am) but then at 11:30 pm the logtext table filled up (the largest table on the site, which holds the text of all journal entries). It hit 4 GB, the maximum size you can address with a 4 byte pointer.

Anyway, we suck. We should have been watching this. We noticed it was getting close a few months ago, but we should've written a script then to notify us whenever a table got near 80% capacity so we could modify it to use a larger pointer size (which is bad to do if you don't need it, since Pentium processors are only 32 bit machines... addressing more than that is slower).

We did the normal maintenance we were going to do, though, and we know a lot more about the MyISAM table format now.

I tried to pass the blame onto Dormando, but that's not fair... we both suck. (hey, I've never claimed to be a great sysadmin/dba. :))

Hey, at least this all happened Sunday night, the lowest LJ traffic time of the week, right?

My apologies to everybody in Europe that was sitting around bored at work on Monday without LiveJournal to keep you entertained. Monday is one of the highest traffic times.

*sigh*

This sucked. Record downtime? I think so.

Oh yeah: Any entry you submitted from 11:15 or whenever the problem started to 11:30 when we took the site down is lost. That sucks even more. :-/

Directory

Yeah, the directory isn't quite working. I know why, though. It's an easy fix, and I could fix it now quickly, but I want to do a few more things and fix it a better way. It should be up in the next few hours, or by tonight at latest.