August 4th, 2001

hacking

Stuff down.

E-mail notifications, amongst a few other things, seem to be down right now. I just woke up, so I'm going to get to work on fixing things now.

Current issues


- Mood icons don't appear, or images don't load.: The servers were moved to a new location last night, and the IP addresses had to be changed. We set up a machine at the old place which is forwarding requests to the old IP address, but it doesn't forward image requests. I'm going to see about getting it forwarding images later today.

Update: Mail seems to be sending now. I'll bet a whole bunch of comment notifications were lost, however :( I can't really tell how much were saved. Drop me a note if you're not getting notifications on comments made after this post was made.

Update 2: Figuring the extent of the broken-ness. I have a growing list of things that need to be fixed, but with any luck I'll be able to restore most of the site within an hour or two. I updated the 500 error message with a short message. It will look more like this message once I fix a few more things.

Update 3: Got the LJ servers syncing code from the new code server, since the old one they were syncing against is now a database. Pulled marklar out of rotation, since MySQL isn't running on it right now. Threw hat into rotation. The site seems to be back now. I'll look into tossing kenny into the pile as well.

Update 4: I've done just about all I can. I'm going to prepare the two new slave webservers for normal use, but I can't put them into the pool since only brad has access to do that. Brad also needs to get the other two DBs going (which should only take a few minutes, after he wakes up.)
Until then, I'll be monitoring the site closely. It seems to all work right now, though. Except for the issue highlighted above.

I'll update this post as stuff happens.

"We kicked ass today."

Yes, Dormando... we certainly did. Damn, what a night.

I think we're just about done with the server move. The redirector machine is doing less and less traffic, which means people's DNS is getting updated and they're hitting the new location directly now.

Instead of being down from 9pm-10pm last night we were down from 11:30-3:30 pm. And then we all went home. I was going to keep working but got tired and fell asleep.

I fucked up a number of things but in retrospect it could've been much worse.

Right now Dormando and I are working on getting all the new servers actually doing work. They're all installed now, but we have to set them up. We have two new webservers running, but 2 databases aren't doing anything right now... that's my job. Right now I'm syncing one up from the other, then we'll put them in the pool.

Now that we have space I think we'll just keep buying machines. I'm finally happy about our infrastructure. Plus, it's just going to get better. This rocks.

If things have sucked for you, hang on... in a few days if not sooner it should be just perfect.

Oh, before I forget--- I'd like to thank everybody that helped out last night:

evan --- you have way more patience than I do installing those damn racks. plus, I would've never figured out minicom.

sherm & katebate -- for helping carry stuff and mounting things.

chuck & xb95 -- for helping transport & mount servers, sticking around until the end, and testing that the phone system at internap works fine, confusing the security guards. also, thanks to chuck for carrying all the heavy stuff, you big man you.

dormando -- for fixing all the stuff I broke. :-)