Things are getting fast, but it's going to be getting crazy fast.
The memcache is done and up and running. For now we're starting light and only have 13 GB of memory used for cache. We'll expand that over time to increase/sustain cache hit rates. Currently the hit rate is 80-85%. (which means the databases are only doing about 20% of what they were before...)
Now we're working on the LJ code, making it uses the memcache more. The most popular page type (S1 recent) we've got down to a minimum of 1 database query, but we'll be removing that. Other pages are being optimized in the same way. Pretty soon most pages will be served entirely from memory. (and even our webservers are diskless, so now disk movement throughout the entire backend...)
The one thing I haven't memcache-enabled yet is friends pages, which is the big DB-killer. I want to re-work a lot of code in that area yet (including making bookmarkable URLs, killing ?skip= and replacing it with ?before=<date>), so I'm going to do that all at once. Anyway, when that's memcache-enabled, that's when things become crazy fast. (You'll be hitting reload a few times, thinking there's no way it could've been that fast, thinking you must've loaded it from your browser cache, and you want the new version, but you'll be getting the new version, and each of your reloads won't ever hit any database.)
We're ordering a bunch more webservers, since that'll be the next bottleneck. (it's already started to become one during peak hours, now that dbs are fast)
We're adding a bunch more monitors, to detect when performance starts to slip in any area.
We fixed our async job system, so pinging weblogs.com and updating /stats/latest.bml works again.
We're putting up a new DB cluster in the next few days, even though it's not really necessary anymore, with the memcache. But it will be eventually. (yes, we're finally getting ahead of ourselves!)
We also need to get a new switch, since we just filled up one of our four.
Plea for cheap hardware:
If anybody out there works for a hardware vendor that can get us a good deal on switches or web servers, email lisa and me. On the webserver front, we like 1U (half-depth, preferrably) dual P4 Xeons with 1GB of memory, PXE bootable, and no harddrive, cd-rom, or floppy.
I'm finally happy with things. Even though it's not perfect yet, everything from here on out is trivial (memcache-enabling code, adding servers) and I'm totally confident it'll be rocking in no time like you've never seen.