bt (dwell) wrote in lj_maintenance,
bt
dwell
lj_maintenance

Network Maintenance - Saturday, October 31, 2009 at 04:00-05:00 GMT/UTC

EDIT: If you're reading this, our maintenance is OVER! The problem was not found on our equipment, which means we'll have to work with our ISP to fix this small problem -- which also means another maintenance window in the future -- but at least we have eliminated our side.

Thank you everyone, and a special shout out to rekoil for giving me a great suggestion AND also the opportunity to feel like I've just called in to a local radio station.

Have a great day, night or afternoon wherever you may be.

---

Hi everyone, sorry for the late notice but I'm going to have to do some testing on 1 of our 4 internet circuits TONIGHT; Friday night or Saturday morning depending on which time zone you're in.

Most of us shouldn't notice any impact, though there may be some slowness or lag when I switch traffic on to our other ISP circuits and then another hit when I stop the tests. If a page won't load or times out, try hitting refresh 1 or 2 times and it should load then. If it doesn't work at all... trust me, I'll be typing really really really fast to try to undo whatever I just did. Hopefully you'll have some Halloween candy (if you're in the USA and celebrate that kind of thing) nearby to take away the bitterness of a small site outage. :(

Here's the handy-dandy Website That I Always Use to get a feel for when the maintenance will start in your area. Our site traffic historically dips on Friday afternoons until Saturday morning which is why we tend to pick this time for maintenance work.


We've been getting low receive power on our "ISP Q" OC12 interface for a little bit of time now. I want to put this interface in loopback and run some hardware diagnostics to see if it's a .. I dunno, "bad" SFP? My feeling is it's either a barely kinked fiber cable or "ISP Q" needs to bump their output on their end since it's a bit of a run from the MPOE to our router. Since I'm not an expert on these kinds of problems though I'm going to do my due diligence and go through all the steps.

I'm going to try to nicely drain all traffic on "ISP Q" by announcing a larger prefix (inbound) through "ISP Q" and and locally preferring outbound routes through "ISP V". Or I could just shut off BGP peering with "ISP Q" right from the get-go since it'll go away when we put the interface in loopback anyways. I suppose in-flight data could be lost but all these protocols are supposed to recover so I really think this is more of a "hey, we're messing with production equipment, there's always a small chance you'll see problems" rather than a "the system is down down down" alert.


status.livejournal.org will, of course be updated before and after the maintenance window. Or else marta will get mad at me. :D

bt
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 52 comments
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →
Previous
← Ctrl ← Alt
Next
Ctrl → Alt →