Brad Fitzpatrick ([info]bradfitz) wrote in [info]lj_maintenance,
2004-05-17 19:54:00
  • Previous
  • Memorize
  • Share
  • Next

20 servers down....

1 of our 4 racks of servers just lost power, so 20 servers are offline.

Unfortunately, users on the Madcow database cluster are temporarily off the net, since both Mad and Cow (which mirror each other) were in the same rack. (We should fix that!)

In the meantime, the site's just kinda slow because we lost most our new webservers.

Will post more when we know more.... Lisa and the Internap engineers are looking into what happened.

  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    switch

    Your reply will be screened

     
    Help
  • 285 comments
Previous
← Ctrl← Alt
Next
Ctrl →Alt →

[info]pat_rice167

May 17 2004, 20:01:01 UTC 8 years ago

sorry about that....was wondering why it was slow tho...

[info]patriot

May 17 2004, 20:03:46 UTC 8 years ago

That explains the problems with the site being slow and coming up unable to find web page.

Twenty servers down sounds like a pretty tall job for you guys.

[info]nrs135

May 17 2004, 20:04:11 UTC 8 years ago

Yikes, a whole rack. Thanks for keeping us posted on this, hope Internap figures it out soon.

[info]altena

May 17 2004, 20:04:21 UTC 8 years ago

Ah, I was wondering why I was having difficulties.

No worries, though! I'm about to go to bed, so you have approx. 8 hours to get things back up to speed. ^_^

[info]warloc

May 31 2004, 11:52:38 UTC 8 years ago

hi

i was just wondering to see if you have actually seen chobits i think its a cool series so that's all


laters

[info]altena

8 years ago

  • Loading...

[info]eddiewoods

May 17 2004, 20:04:35 UTC 8 years ago

Yea I'm on Porkchop and I get "Page cant be displayed" after i check my friends page and I have to refresh to get it to load...

[info]ex_shattered767

May 17 2004, 20:09:34 UTC 8 years ago

Double check where you are.

I thought I was on Porkchop too, but I guess I was moved to Madcow within the last month or something.

[info]bleything

8 years ago

  • Loading...

[info]ex_shattered767

8 years ago

  • Loading...

[info]bevived

8 years ago

  • Loading...

[info]macros06

8 years ago

  • Loading...

[info]jaddziadax

May 17 2004, 20:04:37 UTC 8 years ago

good luck at fixing them!!

[info]dopy

May 18 2004, 05:49:07 UTC 8 years ago

cool icon

[info]jaddziadax

8 years ago

  • Loading...

Deleted comment

Deleted comment

[info]aedgeofeternity

8 years ago

  • Loading...

[info]yum_cimil

May 17 2004, 20:05:00 UTC 8 years ago

You don't suppose it's weather related by any chance? I know we've been getting thunder and lightning storms in the Beaverton area earlier this afternoon.

[info]bradfitz

May 17 2004, 20:10:15 UTC 8 years ago

Internap's in Seattle, and weather's not an issue. That place should be able to take an earthquake that would otherwise level Seattle.

I think we just didn't have enough power in there, and had no way to know. (We don't get graphs or data about that, and we've never planned for it.... just always assumed we had enough power. :-/)

[info]abates

8 years ago

  • Loading...

[info]damnitnicole

8 years ago

  • Loading...

[info]expanding_soul

8 years ago

  • Loading...

[info]damnitnicole

8 years ago

  • Loading...

[info]j9luvsmozart

8 years ago

  • Loading...

[info]risen_phoenix

May 17 2004, 20:05:24 UTC 8 years ago

Damn...

Best of luck, guys.

[info]ex_shattered767

May 17 2004, 20:06:44 UTC 8 years ago

Was wondering why I was having problems. Last time I checked (over a month ago) I thought I was on Porkchop, but a check 30 seconds ago revealed I'm on Madcow.

Thanks for letting us know. Good luck in figuring out what went wrong.

[info]pointclickkill

May 17 2004, 20:34:22 UTC 8 years ago

How do you know which server you're on? Probably a ridiculously stupid question, but I don't have a clue... =/

[info]ex_shattered767

8 years ago

  • Loading...

[info]coolhappysteve

8 years ago

  • Loading...

[info]ex_shattered767

8 years ago

  • Loading...

[info]coolhappysteve

8 years ago

  • Loading...

[info]dopy

8 years ago

  • Loading...

Deleted comment

[info]ex_shattered767

8 years ago

  • Loading...

[info]madmade

May 17 2004, 20:07:24 UTC 8 years ago

So that's why lj is sucking for me.

[info]sunsplash

May 17 2004, 20:08:56 UTC 8 years ago

I knew I shouldn't have purchased a paid account today! I whammy'd it for everyone! Hah! :)

Hope my $25 helps keep lj up and running :)

[info]damnitnicole

May 17 2004, 21:07:08 UTC 8 years ago

OMG u b0rked LJ.

*stabs*

[info]chrishaas

May 17 2004, 20:09:40 UTC 8 years ago

Thanks for the explanation.

[info]kaimi

May 17 2004, 20:11:13 UTC 8 years ago

THAT's what happened....

[info]devlindreadwood

May 17 2004, 20:11:25 UTC 8 years ago

Was wondering why my client had a hard time connecting and i couldent access pages for the worth of dirt

[info]ooglie_wooglie

May 17 2004, 20:11:54 UTC 8 years ago

I think I'm GREEN but I couldn't access anything for about an hour either.

Good luck!

[info]arikia

May 17 2004, 20:12:23 UTC 8 years ago

So that's what was going on. I kept getting the "page cannot be displayed" and I couldn't log out of my other journal.
Thanks for the heads up. Good luck on fixing it!

[info]finalconnection

May 17 2004, 20:17:53 UTC 8 years ago

hah i got the same thing too...i thought it was my computer

[info]onedarkroom120

May 17 2004, 20:12:38 UTC 8 years ago

i was posting comments as it happened, lol

[info]poeticseduction

May 17 2004, 20:13:09 UTC 8 years ago

anyone have any idea of when it's going to be working again?

[info]ex_shattered767

May 17 2004, 20:15:43 UTC 8 years ago

"Will post more when we know more...."

I think they need to find out what went wrong before they can guess how long it will take to fix.

[info]wonderland6

8 years ago

  • Loading...

[info]plausible_alibi

May 17 2004, 20:15:21 UTC 8 years ago

Thanks for keeping us informed.

The site was really slow for me, and I couldn't do anything. I was beginning to wonder why my DSL was all of a sudden slow, only on Live Journal. Ha

Hope you get the problem fixed as soon as possible. ^_^

[info]ckd

May 17 2004, 20:16:20 UTC 8 years ago

I hope you find the lost power soon!

More seriously, that's a real problem (one we've had in my server room at work too) and I hope you find out what went wrong relatively quickly...I'm sure you will, though.

[info]imdwalrus

May 17 2004, 20:17:13 UTC 8 years ago

My journal seems to be working about half the time...Which half is down: the Pork or the Chop? Or does it not work that way? :)

I could've sworn that Pork and Chop were seperate, though, just like Mad and Cow...

In any case, good luck at fixing it up.

In other news, this post is an experiment: will I get any replies at all if I use Lakitu as my icon instead of the now-infamous Duck Hunt duck? :)

[info]schnee

May 17 2004, 20:21:41 UTC 8 years ago

You'll get a reply from me just because you used Lakitu. :) Yay for vintage gaming!

[info]imdwalrus

8 years ago

  • Loading...

[info]mayhap

8 years ago

  • Loading...

[info]pavane

8 years ago

  • Loading...

[info]bugtilaheh

8 years ago

  • Loading...
...and 8 more comments... from [info]astroasis, [info]bossjock77, [info]ex_noda654… Expand

[info]astroasis

8 years ago

  • Loading...

[info]moogleray

8 years ago

  • Loading...

[info]lizzie9208

May 17 2004, 20:18:58 UTC 8 years ago

ah, that explains why the site is so incredibly slow tonite.

[info]finalconnection

May 17 2004, 20:21:31 UTC 8 years ago

its TERRORISTS i tell ya!!!

*spaz attack*

[info]dopy

May 18 2004, 06:28:20 UTC 8 years ago

ha! ha!

[info]crschmidt

May 17 2004, 20:22:14 UTC 8 years ago

Impressive that the site stays up. I think it's a testament to how well you guys have things running that you can lose 1/4th of your servers and still do just fine, albeit a bit slow.

Congratulations on such a great site.

[info]bradfitz

May 17 2004, 20:53:26 UTC 8 years ago

Honestly I was a bit shocked.

I think we just got lucky, though. I'm not sure we could've survived as well with some of the other cabinets gone.

But if we did this well, I think total power loss in a cabinet is an almost-achievable goal... I want to rearrange some stuff now.

[info]crschmidt

8 years ago

  • Loading...

[info]zarhooie

8 years ago

  • Loading...
Previous
← Ctrl← Alt
Next
Ctrl →Alt →
Create an Account
Forgot your login or password?
Facebook Twitter More login options
English • Español • Deutsch • Русский…