Brad Fitzpatrick (bradfitz) wrote in lj_maintenance,
Brad Fitzpatrick

Summary of problems

-- db replication is behind. this causes things to seemingly work and not work over time, like changes you make. i really don't want to explain this all again. paid users don't have this problem, since you guys are only hitting the master database, not the slaves which can't keep up lately.

-- we're readying a new slave that should be a lot faster, running on innodb. if it proves successful, all slaves will be converted. the clever ones out there may be realizing: that means you have two slave dbs right now not doing anything. yup. and theortically, we could make them both serving requests to free users now to make things better, but that's only a temporary solution and we've done in tons of times in the past. it's time to find something that works for a longer period of time. innodb scales, myisam doesn't. time to master all facets of innodb.

-- people with tons of friends are having problems with friend views. avva rewrote the friends view code, but his algorithm sucks for people that have few friends. i cleaned and combined the two, detecting which to use, but it's not live yet. in the meantime, if you want to help yourself and the rest of LJ, remove people from your friends list who post a lot but don't post things that you can see. the current algorithm running loads your friends' posts in the order since they last updated, whether or not they're visible to you. this is a heuristic only, and it usually helps a bunch. sometimes it doesn't, and you end up loading a lot of stuff. we used to load it all in one killer 5 table join. myisam hated that. innodb should love it. and we'd benefit then also from a new index on the log table. we're experimenting with that too.

-- we increased the speed at which free users can reload (well, see new stuff when they reload) from 30 seconds to 60 seconds. if you're reloading more than that, you're not helping. we resisted the urge to make it 5 minutes.

Things aren't going to be great for the next few days for free usres. Everything we're doing lately is on test machines. In other words, stupid questions like "u say u r working on it but the site is still not working why help me" just piss us off. (as well as comments from smart people that assume we're stupid... you trying running this site, thanks.) I don't want to come off as an ass (probably too late) but just understand how very stressful it is. As such, comments are off here until all our new code is live, the new db slave is up, etc, etc.

Comments for this post were disabled by the author