This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

NCM 7.9 Baseline

The NCM 7.9 changes to baseline have negatively impacted how we report on configuration conflicts for audit reason.   I'm hoping this community will review our current usage of Baseline and Compliance Checker and help me clarify how the new changes to Baseline should be applied.

Here is our process for using NCM, Baseline, Config Compliance Checker, Running vs. Startup Config Conflicts, etc., when answering to our Internal IT Audit team:

At the start of  a Quarter we run a job to "Baseline Entire Network".  And any given day, we could go to Config Summary and monitor "Baseline vs. Config Conflicts" or "Overall Running vs. Startup Config Conflicts" to see if and how many conflicts exist. 

If at any time an auditor would inquire about changes or we reached the end of the quarter and had to report on changes we could gather "Baseline vs. config conflicts" from SolarWinds and match them, one for one, with Help Desk Ticket Change Requests.  Then we would baseline all of the configs so that our Baseline vs. Config Conflicts chart would go all green again.

Also from Config Summary we report on Config Compliance Checker Policy Violations.  There is a Report for each designated type of device (access switch, router, firewall, etc.).  This report has a tailored policy consisting of several rules that use Regex or simple pattern matching to snippets of code that are relevant to features for that device type.  And daily this Config Compliance Checker can be monitored for "out of compliance" devices.

But the latest changes with Baseline have me confused on where Solar Winds is ging with Config Managment.  For one, they have taken the Job Type "Baseline Entire Network" away.  That breaks our current process of Baselining everything periodically.  Also, I'm still trying to find out why I can't promote any configs to Baseline, even on a one for one basis.  I thought I had full admin rights but maybe not.  And it seems to me the new Baseline features of ignoring lines or creating baseline snippets or applying baselines to multiple devices of the same type really just bleed into Config Compliance Checker Policy Violations.

Its like Baseline from NCM 7.8 and older is dead as we know it and Baseline will be replacing Config Compliance Checker? 

  • The scenario you described with the legacy baselining is an exceptionally rare situation.  It appears to me that you guys actually built a process very closely around the capabilities of NCM, so it is obviously a pain if the tool changes.  In the field I would say 98% of NCM owners just ignored the baseline feature completely because it wasn't well suited to what most people needed to do.  Given that uptake was poor the NCM team seems to have tried to align the new baseline workflow with the kind of feedback that people were giving them, one being that the all or nothing baseline violation thing didn't make sense for the majority of them in environments with configs that are continuously moving forward.  Nothing you described in your old workflow sounds impossible now, but the specific workflow you had and the specific reports you used will likely need to be reworked to align them with the new features.  I do think you are correct in that there is a lot more overlap now between baselines and policy violations.

    I'd say this is a good argument for testing out new releases in a lab/eval before deploying them to the prod instance, so you can see where changes might impact your existing workflows and you have time to plan out how you will handle the change.  Probably a good idea to revisit the decision process for the old work flow, how much of that was actually driven by your requirements from the auditor and how much of it was driven by the way the tool worked at that time?  I suspect you can probably cook up some new reports that give you the same level of visibility into changes as you had before, but it might take some time to work it all out.

  • Hello.

    Using baselines to keep an eye on changes to the configuration is still a working feature of the latest NCM version. Here is an example from my lab where I promoted one config to be the baseline. The system found and displayed it correctly that there is a mismatch in the config lines and hence my widget "baseline vs config conflicts" shows 100% conflict as I only have this 1 device.

    pastedImage_4.png

    pastedImage_1.png

    Could you please check out this guide to creating baselines as it also lists the permissions required doing so: Establish baselines as a comparison point for config changes - SolarWinds Worldwide, LLC. Help and Support

    Best regards,

    Steffen

  • The workflow decision process was a compromise between what the auditors wanted and what the tool could provide.  I guess ultimately I liked the "all or nothing baseline violation" as a means to quickly summarize what percentage of configs changed from one day to the next no matter how insignificant the change.  That combined with Config Compliance Checker gave us the assurance that no one was tinkering with our configs.  

    Regarding your statement "...nothing you described in your old workflow sounds impossible now..."  They got rid of the ability to use a job to set Baselines.  That seems impossible now.  And as you said configs are "continuously moving forward" which is why we periodically used a job to "set as baseline" in mass. 

    With NCM 7.9, we'll have to change our workflow to "Promote to baseline" for full configs on a device by device basis and if we schedule a config change across many devices at once, we'd have to manually "promote to baseline" for all those changed configs..unless we adopt the new baseline features, thus ignoring more lines of code (not the direction I want to head) and creating baseline snippets that are relevant to many devices of similar type (again, this sounds so much like Config Compliance Checker I'm worried they are going to change that feature next.)

    Is there a roadmap of NCM changes to Baseline and Compliance Checker?  It seems they are going to morph into one big feature set with newer versions of NCM.

  • Yes I agree with you that NCM 7.9 can still keep an eye on config changes but lets say you have 400 devices in your lab and you make a change to all 400 devices.  Did you then have to manually promote 400 configs to baseline?

    There used to be a job you could run to set as baseline in bulk. 

  • Anthony - the optimized workflow in the feature now allows you to promote/create one baseline that applies to 400 nodes and not have to baseline 400 nodes with 400 individual baselines. Do any of the nodes you monitor have similar/same configs?

  • Sure I have groups of like devices...lets say I have two hundred routers that are similar in machine make/model and even similar in purpose (Branch Routers that connect to MPLS service provider.)  But things like hostname, network statements, IP addresses, port descriptions, ACLs, etc, that make each config unique to that device and have everything to do with how that device functions, would all have to be ignored if we use a single baseline for all 200 of those devices.

    And what if someone makes a change to an interface, or they inject a network into a routing protocol  or they change some other line that I had to "ignore" in order to get that config to comply with the new baseline?  I'd never know about it. 

    To me the feature you suggest we use is exactly what we are doing with Policy Violations Config Compliance Checker.  So now I'm afraid that Baseline and Policy Violations Config Compliance Checker will both morph into one and we will be stuck retooling our Config Management process.

  • I agree with Thoar500, this functionality is crucial.  I have almost 800 nodes, spread across multiple hardware types, vendors, and versions of code.  Creating a baseline for each is a huge job that would need to be repeated any time changes are made.  We have some devices, that there are changes on a daily basis, others that don't change except during quarterly maintenance.  Staff is our most precious resource and we are now going to have to devote more time to this.

    Maybe it is time to look for other solutions.

  • I want to agree with Thoar500 as well. We performed a process very near what he describes. We have 1500+ devices that we would do an weekly baseline report on and then baseline the entire network. That way we had an easy way to see if there were unauthorized config or config that was not cleaned up after a troubleshooting event. I'm definitely missing this function. I see no easy way to achieve a network wide baseline right now. Boo. :-(

  • This absolutely broke NCM in my opinion. Each device should have a known good baseline you can keep and not have purged.Clean up jobs previously could always be set to ignore the config selected as baseline.

    There are way too many device types, device purpose, make/model etc. to have such a limited baseline ability.

    What a disappointment in development.

  • To clarify your statement, you are stating that the ability to baseline each node and have individual baselines is the primary missing function now? Is this seem correct?


    I also do not see a feature request for this yet, but that is a great way to have the community vote in support and to help me track the potential impact if it is selected.