cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Configuration management… More than meets the eye

Level 11

I’ll be honest, when I initially saw the words configuration management, I only thought of managing device configurations. You know, things like keeping backup copies of configurations in case a device bit the bucket. However, the longer I’ve been in the IT field, the more I’ve learned how short-sighted I was in relation to what configuration management truly meant. Hopefully, by the end of this post, you will either nod and agree or thank me for opening your eyes to an aspect of IT that is typically misunderstood or severely neglected.

There are several components of configuration management that you, as an IT professional should be aware of:

  • Device hardware and software inventory
  • Software management
  • Configuration backup, viewing, archiving, and comparison
  • Detection and alerting of changes to configuration, hardware, or software
  • Configuration change management

Let’s briefly go over some of these and why they are so integral to maintaining a healthy network.

Most (hopefully all) IT teams keep an inventory of hardware and software that they support. This is imperative for things like service contract renewals and support calls. But, how you keep track of this information usually calls for question. Are you manually keeping track of this information using Excel spreadsheets or something similar? I would agree that it works, but in a world so hellbent on automation, why risk human error? What if you forget to add a device and it goes unnoticed? Wouldn’t it be easier to have software that automatically performs an inventory of all your devices?

One of my favorite components of configuration management is configuration backup and the ability to view those backups as well as compare them to previous backups. If your Core switch were to fail today, right now, are you prepared to replace it? I’m not talking about calling your vendor’s support to have them ship out a replacement. I’m talking about rebuilding that new shiny piece of hardware to its predecessor’s last working state. If you have backups, that process is made easy. Grab the latest backup and slap it on the new device when it arrives. This will drastically cut down the recovery time in a failure scenario. Need to know what’s changed between the current configuration and 6 months ago for audit purposes? Having those backups and a mechanism for comparing them goes a long way.

There are a number of ways to know when an intruder’s been in your network. One of those methods is through the detection and alerting of changes made to your devices. If you don’t have something in place that can detect these changes in real-time, you’ll be in the dark in more ways than one. How about if a co-worker made an “innocent” change before going on vacation that starts to rear its ugly head? Being able to easily generate real-time alerts or reports will help pinpoint the changes and get your system purring like a kitten once again.

In conclusion, configuration management is not just about keeping backups of your devices on hand. It involves keeping inventories of those devices as well as being able to view, archive, and compare their configurations. It also includes being able to easily detect and alert on changes made to your devices for events like catching network intruders. Are you practicing good configuration management techniques?

19 Comments
Level 21

We do basic configuration management in our normal environments, mostly backup based such as that provided by NCM.  We include FIM capabilities in our compliant environments.

Level 15

I liked what NCM was doing for our switches so much that i've cobbled together a simple SAM powershell monifor which reads in a server Config file, could also be xml or even DB tables, diffs it against the last copy saved and if any lines are changed makes a backup of it. Next phase I hope to add in commiting the file into github which can show a visual version diff between changes and even against a known good "master" version.

all this after getting Puppet cert and trying to figure out how to train our engineers in operational config management. If it's not easy it's not going to happen

Level 13

Our config management unfortunately only touches that being the backup side of things. Though coincidently this afternoon a colleague asked for a device config from before a particular change was made 3 months ago because x and y stopped working. Within 5min I had the previous config, as well as a Diff exported and emailed to him highlighting what went astray.
We tend not to do the 'save on commit' as our policy is to keep 20 config versions - If people are 'learning or being over cautious' then there goes 50% of the configs, so an overnight audit is fine.


Config templates and compliance is something we'd like to get using, but it's getting those stakeholders initially engaged and spend the time to plan, so they realise the long term savings automating the checks and report only those we need to fix after raising appropriate change requests...

MVP
MVP

Our network team used to use NCM, I am not sure what they are using today.

In a past life, the company I worked for used NCM to backup all the configs for the network gear as well. 

I agree with your assessment of components...one of the weakest is the configuration change management portion.  Unless a shop is strict about

what detail is placed in the change ticket, there is much that can be excluded from the ticket.  In my opinion, the change ticket needs to detail all the

minutia of what is changing so that anyone with the appropriate authority can perform the change or revert it...  This provides for a complete picture of what has to change to

implement a particular change.

Level 13

The combination of NPM and NCM has helped me quite a bit with all the config management points mentioned above.

NPM's sonar discovery has identified devices that I didn't know were there, often before our project group makes operations support aware of them.  The only issue I've had with discovery is when the new, or foreign, device doesn't have my SNMP community string or ACL applied. 

NCM has been so helpful in keeping all my configs up to date, but it has also been very helpful when I write a report to identify particular device models/versions with a particular config as identified in a bug report.  That output then tells me which devices need a workaround implemented and upgrade scheduled.

As mentioned above, the big caveat to all this is that the devices must have some basic and standardized config before they can be managed and further standardized.

The archive facility in some Cisco gear is another handy tool to use.

This facility can tell you who executed what config changes.  This is especially useful when some change results in a new config backup not being available to compare against the last known good configuration.

Level 12

Let's not forget about Standardization, Compliance, and Reporting.  These are integral parts of the entire Configuration Management spectrum.  NCM adds all of these plus previous items in the original post.  I also like the integrated Approval System within NPM which, if implemented properly, would ensure that base device configurations, changes, modifications, etc are coordinated and implemented as required to allow the devices to be monitored, managed, copied, backed up, and more from the get go.

Great product and yes, I know there are others out there, that even I have used in the past, but add to all of the above integration into a single-pain-of-glass management capability that is afforded with it being a module that allows integration with other SolarWinds products.

Level 11

I agree bspencer63, standardization is crucial when it comes to configuration management. It makes life easier from all perspectives, especially when troubleshooting. It's one of the first things I focus on when working on a new network. I want the configs to be clean and identical wherever possible.

MVP
MVP

biggest issues for are support for the newer hardware or vendors. That is where the price/performance is most attractive

Level 9

kmillerusaf Enjoyed reading this article.

"How about if a co-worker made an “innocent” change before going on vacation that starts to rear its ugly head?" - I can't tell you how many times this has happened, innocently or intentional...

MVP
MVP

that is what a former team I was on referred to as the "Waller effect".

We has a person whose lastname was Waller make a undocumented change to the remedy ticketing environment on a Friday afternoon and then go on vacation where he had no phone contact. 

It broke the ticketing system and no one knew what changed to back it out or how to fix it.

Level 11

Yep, that's exactly what I was referring to and it's bitten me a few times as well. You end up spending an inordinate amount of time reverse engineering someone's changes and trying to think like them. It sucks.

Level 13

Wish I could say I've never done that.

Lesson learned: don't make firewall changes the day before going on paternity/maternity leave.

MVP
MVP

Using NCM to compare configurations has come in handy a lot of times. Similar to squinsey above, when an engineer comes up with a request for config changes that have been made in the last day/week/month and you can provide that information readily? Makes life a lot easier.

And the times when I get an email about a change made to a device and go "what the... ".

Level 14

We do a pretty good job of configuration management, although some of it is still manual.  We still use some spreadsheets.  We have automated a lot of it.

MVP
MVP

We use NCM to download configs.  To the database and filesystem of each poller.  A scheduled job runs daily to make a list of each config from the day (by traversing the filesystem).  That list is then used as an argument for an SCP job to copy configs to another server (via scheduled job).  The configs then live in the database, the pollers' filesystems, and the server they were copied over to.  Additionally available via backups of the database,backups of the filesystems of pollers, and backups of the filesystem of the server that SCP job copies to.  An additional step in the batch file job on each poller is a count of the number of configs for the day.  This helps in catching config download issues (if we see the number of configs trending down).  SolarWinds support also provided a SQL query to break down the number of downloaded configs each day, and we use that.  If only this was baked into NCM natively.

Level 17

We use the NCM for a lot of these, it's the bottom of the list that needs a slight improvement - currently we take a regular inventory or hardware/software and modules/cards (even custom reporting shows this a few different way since we are so Big), Configurations and Inventory reports and jobs are staggered within NCM to help balance the load.

  • Detection and alerting of changes to configuration, hardware, or software   - it's the real time alerting that needs fine tuning, or turning on. - We have alerts and I can see who makes change, but I need to dig to find out what. And if anything adverse changes I would have to go through infosec to get that stream of traffic and details that came through the Border/Edge and then through what nodes did they travel through...
  • Configuration change management - better base lining needed to get a clearer picture of policy violations / incorrect setups - but DL and viewing changes is in place with no issues.

The inventory via NCM and jobs to keep it updated have helped replace the old spreadsheets. Though I think with the Web Reporting Ability We give them the real time data with the option to export and have their 'Spreadsheet' for reference... some folks just like to live in the past

Level 11

We do a pretty good job with configuration management it really comes in handy when you have to repeat system builds, troubleshoot system, application or network problems.  Having a good CM process will save you a lot of time on the back end.

Level 15

The only drawback my organization has is NOT having a good change management process.  Overall, the configuration management is in check but things can and do get modified without the supporting documents.  The recent staff turnover has brought some of that to the light and we are actively engaged in solving that issue.  Great post!

Level 11

Thanks jkump‌, I appreciate that.

About the Author
Co-creator of Thwackman! I have been in the IT field for the last 17+ years, specializing in networking. I have supported numerous DoD IT environments and networks throughout the world. I am well-rounded in all areas of IT, to include Networking, Information Systems and Services, and Information Assurance. I am currently supporting the Air Force as a Senior Network Engineer on an IT Integration team. I evaluate, recommend, design, integrate, configure, and test new products for our environment. I also develop all documentation and drawings for this entire process. I work with a lot of new technologies. I do a lot with virtualization, network storage, networking, IA, NMSs, SQL, and SolarWinds customization. I have designed, implemented, integrated, configured, documented, tested, and evaluated the SolarWinds Network monitoring/management platform that is currently being used internationally by one section of the Air Force. I have designed SolarWinds VM templates that are used to deploy a preinstalled, almost turn-key solution. SQL tables are pre-populated, Registry configurations are preconfigured, website settings are preconfigured, map templates are prestaged, etc. The installer runs the wizards, follows instructions, and inputs variables for their environment. I have designed, documented, implemented, configured, tested, and evaluated Cisco/Windows 2000/2003/2008 Active Directory/DNS/Exchange Server and Cisco VoIP Infrastructures for 50+ site worldwide WAN networks. I have held many IT postions and titles, to include Information Technology Specialist, IT Consultant, Technical Lead, IT Manager, NOSC Lead, Network Engineer, Systems Engineer, Lead Engineer, Installer, Network Technician, Network Administrator, Systems Administrator, NOC Administrator, HelpDesk Technician, and PC technician. I have developed thousands of pages of documentation (training plans, standard operating procedures (SOPs), integration plans, install guides, troubleshooting procedures, schedules, diagrams, etc.) and have trained 100+ personnel in IT areas. I have a Bachelor’s degree in Computer Information Systems and am currently working towards a Masters in Network Management. I recently served as a Manager of 35+ IT personnel across 6 different departments. I hold many certifications and also instruct EC-Council CEH, CompTIA Security+, Network+, A+, and Cisco CCNA courses.