Brad Fitzpatrick (bradfitz) wrote in lj_maintenance,
Brad Fitzpatrick
bradfitz
lj_maintenance

more work

Paid user requests are at fast sub-second speeds, but free users requests just got behind at around 3-4 seconds because our one old database cluster that hasn't been upgraded over the past month is now croaking in load.

SO .....

We're gonna work on it now, instead of at night.

There will be two steps:

-- Step 1: the users on Tbone Cluster (Where am I?) will go read-only for like a minute while we convert it from master + 2 slaves to 2 masters. (so basically we're making the two current unused children be each other's masters... and the active parent now will be free....) The 2 new co-master machines are beefier than the old parent. All good.

-- Step 2: we'll take that old hardware that was running Tbone and make it help out the Chef cluster which is the last unconverted cluster. (everybody else is InnoDB now, except Chef that's still MyISAM, and straining). That step will be easier than step 1, but both are relatively easy.

Just letting you guys know why your accounts might be read-only for a bit... it'll get rid of the 3-4 second delay for free users.

Then over the next week we'll be slowly moving all Chef users to other clusters that are already fast and InnoDB, then Chef hardware will be upgraded and repurposed.

Got all that? :-)
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 249 comments
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →