cancel
Showing results for 
Search instead for 
Did you mean: 

Network Change Management - Disaster struck, now what?

Level 10

Do you treat network configuration management with the same care and love as you do for server backups in your environment? Or, are both equally as unattended to as the other? No, this is not a post about backups, that would be too boring right? Well, I would be willing to bet that some environments treat network configuration management as low down on the priority list as backups. Let’s start with something as simple as backing up the configuration of a network switch or router. Are you thinking, “Well now why would I ever do that”? Let’s be hopeful that you are not one of those organizations but rather one who could answer with “we backup all of our routers and switches nightly”. If you do not back up the configurations of your network devices, how would you recover in the event of a device failure? Hopefully not stuck trying to remember what the configuration was and having to rebuild a replacement network device from scratch. That would be way too time consuming and guaranteed to be a failure. Now, what about tracking changes to your network devices? Is this something that you keep track of in some sort of change management solution or at the very least even to a central syslog repository? This way you could be alerted when a change occurs or have the ability to identify a change that may have been the cause of a network outage? Having a good change management solution in place for network devices should absolutely be crucial to any environment. Without one, you will be left wishing that you had one WHEN disaster strikes. Now, how do you handle change management in your environments?

12 Comments
Jfrazier
Level 18

While I am not on the team that manages network configuration management, I do know the configs are backed up nightly.

I know they also track changes both through their tool and change management...both go hand in hand.

clubjuggle
Level 13

I've been a user of NCM since the Cirrus days. In my experience I have found nightly configuration management and real-time configuration tracking to be tremendously valuable, and irreplaceable in the event of equipment failure or a critcal configuration error.

zackm
Level 15

This is one of those few places in life where I wish people would just assimilate... (One of Us)

With the tools available, and the relatively TINY file sizes you are placing on storage, there is just not a reason to NOT back things up. Yet I see it so often that it's mind boggling.

Perhaps it's just that before a company invests in a tool like NCM, it just wasn't as feasible? I'm not sure. I do know that, coming up though network engineering, we were not allowed to even submit a change request until we could check off that little box that says "Backups Performed?"

jkump
Level 15

That's my environment here.  No changes submitted until proof that a backup exists.  NCM makes the task of backing up configs painless and we perform weekly audits to ensure everyone knows what changes have been made and when. 

cahunt
Level 17

We backup nightly, so as long as changes are documented, and of course approved you just stay within your change window.

  and our backups have backups

zackm
Level 15

backup-ception.

clubjuggle
Level 13

Proving a backup (or weeks or months worth of them) exists is SO much easier when you have something like NCM...

cahunt
Level 17

This is true - NCM has saved my bacon at times..

clubjuggle
Level 13

Sounds like you're confusing it with Solarwinds Sandwich Monitor!

cahunt
Level 17

Sandwich Monitor Protects my Bacon in Real Time!  I always enjoy un managing my sandwich for consumption.

tcbene
Level 11

‌NCM makes it so easy.  Set a baseline and backup nightly.

mrlesmithjr
Level 10

It is good to see that there are actually some who take backing up network configurations serious. Anyone use any open source tools?