June 5th, 2003

  • lisa

Maintenance event

We are performing some work on our load balancers that should not cause downtime but may affect services briefly. We will update here when finished.
  • lisa

Downtime update

In what was supposed to be a routine security upgrade, the new BIG-IP software from F5 resulted in a comedy of errors, including the last few hours of downtime.

We've been working with some competent people there and they've been working on a fix. Supposedly this bug has affected other sites as well.

As the machines were already rebooting several times a day, we felt there was more urgency in fixing the problem as soon as possible and that this downtime would be brief. We were wrong. The good news is our redundant load balancer is now in production.

In all fairness, we haven't had any major problems with our BIG-IPs over the past year or so, and F5's been working hard to isolate and fix this problem.

A temporary solution is in place now, and the real fix should be available to us tomorrow.

To be clear, this downtime is not related to the distributed memory cache work we've been doing lately. That's proceeding wonderfully, speeding up the web and database servers, as mentioned in Brad's last update . However, if both of the load balancers are down, you wouldn't be able to reach anything on our site.

Please accept our very sincere apologies.