We found that thttpd was hogging all the CPU on a few free web slaves. I've had countless frustrations with thttpd, so we removed it from all machines and we're now using TUX, which is amazing.
Other issue is that we hit a limit on Jesus (master db server), but that was solved by upping a limit in the config file and restarting it, which only took a second or so. That was the bigger issue. Although it was easy to fix, it took awhile to find it.
sherm's at the NOC now installing one db server he repaired and trying to fix a dead web server.Things should be smooth tomorrow. I'm going to be at the Olympics from Tuesday to Sunday (free tickets! :P), so I hope to get the site in a good condition for
dormando and Sherm while I'm gone. 