We've been working with some competent people there and they've been working on a fix. Supposedly this bug has affected other sites as well.
As the machines were already rebooting several times a day, we felt there was more urgency in fixing the problem as soon as possible and that this downtime would be brief. We were wrong. The good news is our redundant load balancer is now in production.
In all fairness, we haven't had any major problems with our BIG-IPs over the past year or so, and F5's been working hard to isolate and fix this problem.
A temporary solution is in place now, and the real fix should be available to us tomorrow.
To be clear, this downtime is not related to the distributed memory cache work we've been doing lately. That's proceeding wonderfully, speeding up the web and database servers, as mentioned in Brad's last update . However, if both of the load balancers are down, you wouldn't be able to reach anything on our site.
Please accept our very sincere apologies.