I think a wise man once said, "you better check yo'self before you wreck yo'self" or something along those lines. ONTD is great because not only do they provide hours of fun, they also push our application and highlight the pressure points. No, ONTD didn't break LJ, and no they didn't break themselves (well, not really), what they did is point out that "hey! there might be a journal or community that has over 16 million comments! no, really!".
Here's the poop.
Problem: LJ was designed with a 24 bit limitation on number of comments -- in the database and in crucial parts of our commenting code. This may have been done for speed optimization, resource conservation, for the children, whatever. It was done with good intentions and now we have BAAAD intentions because the theoretical limit for a journal's comments is 16,777,216 (2^24). And ONTD just did *that*.
Solution?: Our crack team of crack teamsters is picking through code and releasing the 16 million limit (see here) as well as modifying the databases.
Risk?: We haven't deployed ANY of the new code to the website yet, and we've only started testing the alter statements on one of our secondary databases, but if things continue to test well, there is a SMAAAAALL risk that when we attempt to modify the rest of the databases, LJ could drop bits. All over the floor. But we're really trying to make sure that doesn't happen, and in spite of burr86 and dormando's help (hahaha, got you guys) tupshin, henrylyne, gariev, nicholaskurjan and mhwest will continue to work at getting rabid ONTD fans their due fix. It might be a bumpy weekend.
EDIT@08:16AM UTC/GMT (12:16AM PST): The problem is even more insidious than we initially thought! That means the original ONTD community is going to be in read-only mode until at least tomorrow when a more comprehensive solution can be attained. In the meantime, if you hurry to (the temporary) ohnotheydidnt you might be able to get in your comment before the 60th page. Like I said though, you gotta hurry.
So we hit a limitation on one of our databases and we're going to do some emergency maintenance to move ONTD (ohnotheydidnt, please don't go there because it's uhhhh, overloaded, PLEASE, THINK OF THE CHILDREN!) and well, I'll edit this post when we're done. Again, this is only going to affect ONTD. We're going to put it in read-only mode so no new entries and no new comments until we're done. Thanks for your patience.