Network Loop due to weather or ID10T error?

score 6
Voted on 6 times. You have not voted. Open for Voting

I work for an Electric Coop which also happens to be an ISP, for very rural areas. Since many of our locations are linked together by way of radio links whenever there is a storm it is common for the those connections to go down. So, while this crazy storm is going through our area I did my various network duties keeping an eye on things and configuring switch ports, so the techs can connect some new equipment in the office. Of course right at 5PM, on Friday no less, the network starts to go down at our main office. Now this storm is getting pretty bad as I'm trying to time it right so I can get home without getting flooded out. The storm HAS to be causing it, so I think the best thing to is to just wait it out. My colleagues continue to troubleshoot it though while I head home.

 

Hours later I find that they're still working on it and can't seem to find the cause. It has to be the storm, I keep thinking. Come the next morning and there is still a major outage at the office. I figure I'll remote in and try to take a look. All of a sudden it hits me...what if something went wrong with one of those ports I configured for the techs? I figured what the heck, I'll disable the ports that I worked with them on and see if that fixes it... Network outage clears right up. I call one of the techs and he was certain that nothing was plugged into that port. Come the next work day, I get in the office and after tracing it down I find that a switch was plugged into this configured switch twice...causing a loop. Just so happened to have a security video of what happened and it seemed that after a very long day one of the techs saw a loose cable hanging out of a switch and then proceeds to plug the other end back into the switch on the same stack!  Safe to say they were pretty sorry and just had a major brain fart.

 

Moral of the story is even though the storm was huge and was the most obvious cause, it turned out to be one small mistake that caused a big outage. So, I saved the day....sort of.

Comments

Vote history

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.