January 18th, 2004

  • lisa

Update on Santa cluster outage

We're been working all night on resolving the unexpected problem with sub-cluster 7 of Santa cluster.

Currently, the slave databases on Santa cluster need to be set up again. In order to do this, and to ensure database consistency, we need to take the Santa cluster down again. This will cause identical problems to last night -- users on Santa may not be able to login, post, or have their journals available for other users to view or post in. This time, though, we're doing it deliberately, and our estimated time for the last of the repairs to be complete is by 10AM PST (GMT -8).

We really are very sorry, and we're working as fast as we can.

I'll update here when this is complete.

UPDATE: Once again the cluster is back up. Note that the site overall is slow (and sometimes unresponsive) as the database caches fill up, but will clear up shortly. Please comment here only if you are on the Santa cluster and continuing to experience any login or posting problems. Thank you for your patience.