I was the junior network guy at my previous job and no, this was not a resume generating event. I was working at a DoD facility that did large network centric military exercises. Having over 1000 participants in the facility was considered normal. It was also very common to finish an exercise on a or only to get set up immediately for another one the following . The exercises would make use of networks of different classification levels depending who was participating in the event.
One communications closet that supported 3 large game floors was equipped a fully dressed out Cisco 6513. For the uninitiated, a fully dressed 6513 is a layer 3 switch capable of hundreds of access layer connections. There were fiber connections from the communications closet to the game floors that would be used at the primary network classification level of the exercise. The 6513 would "swing" between different networks in support of the upcoming event.
It was a afternoon at about . An unclassified exercise had just finished and a classified exercise was starting the following . All of the unclassified machines had been disconnected from the switch. This would happen first so they weren't inadvertently put on a classified network. After everything was disconnected, I moved the switches' trunk up link from the unclassified network to the classified network. After getting a link, I walked down to tell our lead engineer that the move had been made.
As I walk into his office, his classified What's Up Gold (Pre SolarWinds) display had gone completely red. Did I mention that it was on a afternoon. One of the switches that went red was nearby, so we grabbed a classified laptop so we could console in. Once we got logged in, we noticed that none of the VLANs on the switch were correct. This would explain why nothing was working. But wait! Those VLANS were the same as the unclassified network! WTH! Thus began my first lesson with VTP (VLAN Trunk Protocol).
For the uninitiated, VTP is a protocol that allows a VLAN created on a "Server" switch to be automatically pushed out to all of the "client" switches on the network. This eases the burden on the network administrator. In lesson one, I learned that the default setting for a switch running VTP is server. I also learned every time the VLAN database updated, the configuration revision number is incremented. This revision number gets advertised to the other switches participating in VTP. If a switches' current VTP revision number is lower than the advertised revision number, it sends a request for the updated VLAN information.
Well, apparently the VTP revision number on the newly attached switch was higher than the one currently in use on the classified network. In the time it took me to walk down to tell the lead engineer what I had accomplished, VTP did what VTP does and all of the classified switches now had the wrong VLAN information, effectively taking down the entire in house classified network. Mentally I am hearing Steve Urkel say "Did I do that?" Did I mention we had an exercise starting ? Armed with this knowledge, the lead engineer sent out an all hands on deck. Yep, now the whole network team is going to get to stay late on a afternoon because of me.
Since this was before we had discovered the wonderfulness that is SolarWinds , we had to manually pull the most current configuration file from backup and console into our primary switch to replace the running configuration. Prior to this we consoled into the offending switch and reset the VTP revision number. We got the network up and running around that evening. That event cost me lunches all around for the team, but I became well versed in VTP in the process. So, all's well that ends well. I lived to fight another day.