We have not forgotten about LiveJournal... We have just been really busy :)
There are a ton of new things hitting the site in the next week or two, these should greatly help with the speed problems. I mean seriously, we are awesome at making things go faster, it is just never enough :)
To note, http://status.livejournal.com is a separate server which will contain emergency information and the like. Go there if you cannot get to lj_maintenance.
Stuff we are getting or new methods we are using:
- More Memory - We just bought a lot more memory for existing servers, which will bring just about all of our database servers to the maximum amount of ram they can hold (about 2G each)
- New Servers On Their Way - Yeah. We bought some four new servers right after buying the last round of four servers. These are a little beefier, and will have a new configuration to make them even faster than before.
- Rebuilding Our Current DB Slaves - Once the new ones arrive and are live, we will be taking the old ones down and tweaking the hell out of them for the new configuration.
- The New Configuration! - RAID 5 on slaves was a really stupid idea. So we are switching them all to RAID 0, which is mucho faster. We do not need to care about data integrity on the slaves, if something breaks, we just resync. We are also enabling write-back on the RAID cards, adding delayed key writes to the MySQL configuration, and switching some tables to a new MySQL table type (InnoDB.) The new table type has fun things such as granular row-level locking and multiversioning. Yum yum yum.
- New Code - We love writing code... most of the time. There are some huge patches which will be going live soon. These change just about all of the remaining code to be optimized for the DB slave usage. Before we were moving just the really big stuff to the slaves, but now we're moving the rest of it over, which should make a healthy difference. Especially with all of the new DB slaves arriving.
The usual helpful tips:
Reload less, it works more often. We can't stress this enough.
Peak hours stretch across 10AM PST to about midnight PST. That covers most of the day, but not all of it.
status.livejournal.com might give you some weird errors. That just means your ISP hasn't caught up with our DNS yet. Give it a few days and the problem should fix itself.
Update: I ran out of candy. I want more candy.