Lisa Phillips (lisa) wrote in lj_maintenance,
Lisa Phillips
lisa
lj_maintenance

Update on missing journal entries

This is an update on the post I made on Monday regarding missing journal entries for users on the ribeye and santa clusters.

Unfortunately after this announcement and the fix of that bug, there were still 314 accounts (out of a total of 228,000 total user accounts moved) which appeared to have had their contents deleted (though there was at no point any actual data loss). These have all now been identified and fixed, and affected paid users compensated with a year of free paid time. We know the root cause of this particular problem was due to the bug on Monday, and don't expect it to happen again in the future.

Our first wave of user moves is now complete. No data was deleted during this move, just copied, and all source data has been backed up. We are restarting all of our memcaches tonight, so there should no longer be bad data cached on our side.

We will start our next group of user moves only after we're confident all is as it should be with this group of users. We will post before we begin the next group.

In good news, Ribeye (the new db) in its new configuration is handling a large amount of users much better than we've seen the performance on previous configurations of our clusters. We look forward to applying the same methodology to the other db clusters and greatly increasing our overall speed and capacity.
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 121 comments
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →