I think it all started with the traffic finally getting to a level such that the half-duplex to full-duplex link was so noisy with errors that the bandwidth internally just went to hell and things got behind.
Once requests start cascading, all is screwed. People start reloading over & over again, doubling or tripling the server load.
There were a ton of minor problems along the way. Most of the frustration was dealing with users that were freaking out... but that's to be expected.
I'm not sure how much longer I can handle this & stay sane. It's always one thing after another. I was having fun actually working on real LiveJournal stuff today and thought I'd get to all night, but then everything started breaking.
I also got a letter from a law firm today saying they're charging us $1,500 in damages for a piece of spam we allegedly sent them. If they'd check the headers they'd see we've never sent any spam.... other people are sending it through their normal servers and then just forging their return address to be @livejournal.com. Swell, huh? I hate spammers. So tomorrow I get to talk to that lawyer and explain how SMTP works. Hopefully he still has the full message, including headers, and we can work things out.
But the point is... none of this is fun.