All is pretty much okay, compared to all the breakage yesterday. Only problem now is that the 3 free user database servers are behind in replication, so posts/comments are showing up lagged quite a bit. As a result, people are posting their stuff multiple times, causing the replication to get further behind.
Things that are happening to fix this, in order on when they're expected to be done:
-- since master db wasn't loaded, i made it handle 20% of reads... since i did that this morning, the free user dbs are slowly catching back up, with the exception of kenny, which just sucks. kenny's only handling a small percentage of requests, but it still sucks. (we're going to rebuild kenny when we have a chance... its raid configuration is still 1 with a write-thru cache from back when we were using it for something else... mad slow)
-- going to prevent some writing (like, dups) that'll ease slaves as well.
-- new memory for db slaves on its way, brining them all up to 2 GB
-- two new db slaves arriving on the 1st
The more controversial thing is that we're probably going to limit the number of posts/day and comments/post for free users. We have to decide between the lesser of the evils --- limits or having a site that always appears to suck.[1]
I already mentioned this in
paidmembers, but one db slave is now doing nothing but paid user traffic, and it's totally caught up... there should be no lagging if you're logged in.[1] Update: Relax, everybody ... we're not "cutting off" free users. Stop freaking out ... we understand why LJ is a successful community (we made it, afterall) and we're not going to do anything to bastardize it. Any "limits" we impose will be reasonable... we always collect stats before imposing limits such that 90% or so of people remain unaffected. The limits we're talking about here are temporary ones, to ease the free db servers until the new ones arrive on the 1st.
← Ctrl ← Alt
Ctrl → Alt →
← Ctrl ← Alt
Ctrl → Alt →