June 5th, 2001

hacking

More server slowness, but there's hope.

Hey all,

There were some rather nasty load spikes on the DB earlier, and I know everyone's getting sick of these error messages... I've been doing what I can while brad finishes finals, which isn't too much longer. There is hope! We're going to get so much done after friday, you'll all be amazed. ;)

I turned off some daily indexing jobs on cartman, which were causing some troubles when they ran.

I also got netsaint running on LJ's network today. This allows us to have more detailed information about every service/host on the network. I'll also be paged with details about a problem within ten minutes of it happening, so I can respond to problems faster, and fix them quicker.

Just think, if I had a life/SO, all of you LJ users wouldn't have at least one 24x7 systems administrator :)

Yeah! Or something. I do believe this won't be a horrible problem, so much stuff is speeding up already, we're in great shape. This won't be as bad as it has in the past.
I'm just begging for forgiveness from more errors, I'm done now ;)

DDoS

We got SYN flooded by a DDoS. 50 Mbps of incoming SYN packets isn't good. Things broke.

We setup a bunch of firewall and rate limiting rules on the BIG/ip now, though. Shouldn't happen (as easily?) again.

Whoever did it is a punk. These little dickless morons can't break into the car, so they decide instead to slash its tires. Well now our tires have metal guards. Die.

I need to go study... two finals tomorrow. 3 days until freedom.

Thanks to dormando, ff, and lisa for diagnosing, helping fix, and rebooting, respectively.

Garrrrh.