Earlier I posted that our bottleneck had become webservers. So we converted one of the DB servers into a web server. I'll do another one tomorrow if needed. (and after that, we'll use one for the D-word)
Another thing I did to improve things was stop using TUX2 (a webserver) since it sucked (as does thttpd) and was eating up all the CPU on two of the web servers on occasion. (I was going to try X15, but had trouble getting it going. So we'll stick with good ol' Apache, since it works reliably. And using it as an image server is okay, since we have light-weight httpd processes anyway (no mod_perl, yet... reviving that port is next project.. it was done once.))
Another thing --- one web server (bebe) looks like it has a failing ethernet port. I'll see if I can get sherm to go plug the cable into its other one tomorrow and see if we can stop the error rate.
Basically, everything from here on out as far as I can see is routine and boring. There are a few more tables on the master DB cluster that I'm going to move to clusters after my final Saturday. And the mod_perl port after that.
But other than that stuff, improving things will just be a matter of slowly buying new hardware as needed, which was my stated goal awhile back. The architecture is now fundamentally sane.
I have a final Saturday and then a really easy one a few days later, at which time I'm done with school forever, which means fun LJ hackin' all the time. So don't bug me too much until then. :-)
(as always, direct confused people to /support/. thanks!)