We're about to mass-move 88,000 users from one database cluster to another.
If you're on the Chef Cluster, subclusters 8 and 9 (Where am I?), you'll be getting "Account is in read-only mode" messages for a bit tonight.
When the operation is complete, you'll be on the Cartman cluster.
Actually, Cartman's a historic name for that cluster... Cartman is long dead (it's our oversized 6U router at the office now). The machines in cluster 1 (Cartman) are "Tender" and "Loin", so it's really the "Tenderloin" cluster. For reference, cluster 8 is "Porkchop". (Machines "Pork" and "Chop").
Tenderloin and Porkchop are our new master-master clusters (where either machine in the cluster can take over at any time) with the insanely fast disks and raid cards.
We're in the process of making all users on the new master-master clusters. We have a new cluster (cluster 9) on the way... it'll be "Madcow". (Machines "Mad" and "Cow".) When it arrives, all remaining Santa and Chef users will move to it.
Then Santa/Chef will be upgraded to the new hardware, probably renamed to be the "Sushi" cluster, and all of the cluster 4 and 5 people (Tbone and Ribeye) will move to Sushi while cluster 4 and 5 are upgraded. Then we'll have an extra cluster, but we'll probably need it by then.
See how we have to shuffle everything around during upgrades, though?
It's totally paying off, though... with the new master-master DB setup we don't have to stress during operating system, database, or disk upgrades... we just switch what half is acting as the master, upgrade the inactive half, and optionally switch back.
Our goal is to get the whole userbase switched over to that config over the next couple months.