Lisa Phillips (lisa) wrote in lj_maintenance,
Lisa Phillips
lisa
lj_maintenance

Downtime update

In what was supposed to be a routine security upgrade, the new BIG-IP software from F5 resulted in a comedy of errors, including the last few hours of downtime.

We've been working with some competent people there and they've been working on a fix. Supposedly this bug has affected other sites as well.

As the machines were already rebooting several times a day, we felt there was more urgency in fixing the problem as soon as possible and that this downtime would be brief. We were wrong. The good news is our redundant load balancer is now in production.

In all fairness, we haven't had any major problems with our BIG-IPs over the past year or so, and F5's been working hard to isolate and fix this problem.

A temporary solution is in place now, and the real fix should be available to us tomorrow.

To be clear, this downtime is not related to the distributed memory cache work we've been doing lately. That's proceeding wonderfully, speeding up the web and database servers, as mentioned in Brad's last update . However, if both of the load balancers are down, you wouldn't be able to reach anything on our site.

Please accept our very sincere apologies.
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 47 comments
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →