Brad Fitzpatrick (bradfitz) wrote in lj_maintenance,
Brad Fitzpatrick
bradfitz
lj_maintenance

Boy did that ever suck

Well, we had planned for downtime tonight for maintenance (at 1:00 am) but then at 11:30 pm the logtext table filled up (the largest table on the site, which holds the text of all journal entries). It hit 4 GB, the maximum size you can address with a 4 byte pointer.

Anyway, we suck. We should have been watching this. We noticed it was getting close a few months ago, but we should've written a script then to notify us whenever a table got near 80% capacity so we could modify it to use a larger pointer size (which is bad to do if you don't need it, since Pentium processors are only 32 bit machines... addressing more than that is slower).

We did the normal maintenance we were going to do, though, and we know a lot more about the MyISAM table format now.

I tried to pass the blame onto Dormando, but that's not fair... we both suck. (hey, I've never claimed to be a great sysadmin/dba. :))

Hey, at least this all happened Sunday night, the lowest LJ traffic time of the week, right?

My apologies to everybody in Europe that was sitting around bored at work on Monday without LiveJournal to keep you entertained. Monday is one of the highest traffic times.

*sigh*

This sucked. Record downtime? I think so.

Oh yeah: Any entry you submitted from 11:15 or whenever the problem started to 11:30 when we took the site down is lost. That sucks even more. :-/
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 111 comments
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →