Brad Fitzpatrick (bradfitz) wrote in lj_maintenance,
Brad Fitzpatrick
bradfitz
lj_maintenance

bad news, good news, bad news, good news...

Lots of mixed good news and bad news. Let's start with the bad, and the theme of the past weeks:

Bad news: the new database servers we ordered over 5 weeks ago still aren't delivered. It's been round after round of chaos and defects for the past few weeks. Very, very frustrating. And because we didn't get this new hardware 3-4 weeks ago, we're massively overloaded. (preemptive answer: deleting old users won't help... old users aren't active, so they're not making the disks move around, which is our problem... the disk heads can't move around quick enough right now)

Good news: paid users can still access the site in under a second. That means our load balancing code to detect paid users and make them cut in line works.

Bad news: free users are having to wait upwards of 10 or 15 seconds! That's horrible. When we designed the free vs. paid balancing code, we were anticipating maximum tolerable delays of 3-4 seconds, which is still pretty ridiculous. 10-15 seconds I consider just broken. Unfortunately there's not much we can do at the moment.

Good news: the new servers are coming sometime. I'm told this week, but I've been told that the past few weeks too, so don't hold your breath. I think we'll also be ordering another set of database servers in parallel with this order, just so we're ready in the future, and just in case the current servers on order keep getting delayed.

In summary--- we're frustrated. We'd ordered those new servers before I left for Germany, before August 13th, thinking we were so ahead of the growth. But then all the manufacturing problems started.

I'm really not trying to push anybody to get a paid account, because I think that's really sleasy, so I won't even link the paidaccounts page, but if you're frustrated with the site speed, a paid account is much faster:

Mon Sep 20 19:53:51 [free users queued: 7713, delay: 12s] [paid users queued: 3, delay: 0s]
Mon Sep 20 19:53:52 [free users queued: 7536, delay: 11s] [paid users queued: 0, delay: 0s]
Mon Sep 20 19:53:53 [free users queued: 7457, delay: 12s] [paid users queued: 6, delay: 0s]
Mon Sep 20 19:53:55 [free users queued: 7422, delay: 11s] [paid users queued: 2, delay: 0s]
Mon Sep 20 19:53:56 [free users queued: 7228, delay: 11s] [paid users queued: 9, delay: 0s]
Mon Sep 20 19:53:57 [free users queued: 7067, delay: 10s] [paid users queued: 3, delay: 0s]
Mon Sep 20 19:53:58 [free users queued: 6865, delay: 9s] [paid users queued: 0, delay: 0s]
Mon Sep 20 19:53:59 [free users queued: 6789, delay: 9s] [paid users queued: 5, delay: 0s]
Mon Sep 20 19:54:00 [free users queued: 7012, delay: 9s] [paid users queued: 2, delay: 0s]
Mon Sep 20 19:54:02 [free users queued: 7435, delay: 11s] [paid users queued: 4, delay: 0s]
Mon Sep 20 19:54:03 [free users queued: 7953, delay: 11s] [paid users queued: 15, delay: 0s]
Mon Sep 20 19:54:04 [free users queued: 8114, delay: 12s] [paid users queued: 1, delay: 0s]
Mon Sep 20 19:54:05 [free users queued: 8459, delay: 12s] [paid users queued: 6, delay: 0s]
Mon Sep 20 19:54:06 [free users queued: 8786, delay: 13s] [paid users queued: 1, delay: 0s]
Mon Sep 20 19:54:07 [free users queued: 8693, delay: 13s] [paid users queued: 3, delay: 0s]
Mon Sep 20 19:54:08 [free users queued: 8464, delay: 13s] [paid users queued: 1, delay: 0s]
Mon Sep 20 19:54:09 [free users queued: 8261, delay: 12s] [paid users queued: 0, delay: 0s]


I guess that's why we made the express lane feature, so paid users aren't penalized when the site goes through rough periods, but I'm really sorry most of you are having to wait so unbelievably long.

I'd say we're working really hard to fix this problem, but we're not --- we're just waiting. Once we get those new servers, though, we'll work non-stop until things are fast again.

Thanks for your understanding.
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 413 comments
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →