The NCM 7.9 changes to baseline have negatively impacted how we report on configuration conflicts for audit reason. I'm hoping this community will review our current usage of Baseline and Compliance Checker and help me clarify how the new changes to Baseline should be applied.
Here is our process for using NCM, Baseline, Config Compliance Checker, Running vs. Startup Config Conflicts, etc., when answering to our Internal IT Audit team:
At the start of a Quarter we run a job to "Baseline Entire Network". And any given day, we could go to Config Summary and monitor "Baseline vs. Config Conflicts" or "Overall Running vs. Startup Config Conflicts" to see if and how many conflicts exist.
If at any time an auditor would inquire about changes or we reached the end of the quarter and had to report on changes we could gather "Baseline vs. config conflicts" from SolarWinds and match them, one for one, with Help Desk Ticket Change Requests. Then we would baseline all of the configs so that our Baseline vs. Config Conflicts chart would go all green again.
Also from Config Summary we report on Config Compliance Checker Policy Violations. There is a Report for each designated type of device (access switch, router, firewall, etc.). This report has a tailored policy consisting of several rules that use Regex or simple pattern matching to snippets of code that are relevant to features for that device type. And daily this Config Compliance Checker can be monitored for "out of compliance" devices.
But the latest changes with Baseline have me confused on where Solar Winds is ging with Config Managment. For one, they have taken the Job Type "Baseline Entire Network" away. That breaks our current process of Baselining everything periodically. Also, I'm still trying to find out why I can't promote any configs to Baseline, even on a one for one basis. I thought I had full admin rights but maybe not. And it seems to me the new Baseline features of ignoring lines or creating baseline snippets or applying baselines to multiple devices of the same type really just bleed into Config Compliance Checker Policy Violations.
Its like Baseline from NCM 7.8 and older is dead as we know it and Baseline will be replacing Config Compliance Checker?
Any updates on this since July? I have just started to implement baselines and find it to be very difficult. I have about 450 devices I am trying to baseline and having to select each one from the baseline implementation window is rather cumbersome. I also was an advocate for the bulk baseline feature previously available.
This absolutely broke NCM in my opinion. Each device should have a known good baseline you can keep and not have purged.Clean up jobs previously could always be set to ignore the config selected as baseline.
There are way too many device types, device purpose, make/model etc. to have such a limited baseline ability.
What a disappointment in development.
To clarify your statement, you are stating that the ability to baseline each node and have individual baselines is the primary missing function now? Is this seem correct?
I also do not see a feature request for this yet, but that is a great way to have the community vote in support and to help me track the potential impact if it is selected.
To set a baseline for a device today I had to log in to the server and find the config. Copy and save it to my local machine. Then import it into the new baseline menu and assign it to the device I needed it to.
Versus just checking the box on a config and clicking on "set as baseline".
This is for network devices... I don't monitor any servers.
Here is how I have been doing it, not sure it is the right way, but it works though I have to bounce around screens. This is after the change control process is done to make sure that the changes are correct.
Already there. What took 5 minutes in the past to baseline all the nodes in our network, now is taking hours. I don't understand the reasoning for removing the jobs, but the new method really eats up my time.
That feature request is specifically more around the dynamic selection once a baseline is created and I think some of the discussion here is to create a job that automatically baselines every node for you (we used to have this feature). We never had the ability to dynamically select which nodes apply to a baseline after a config was flagged as such.
In the case of removing the job to baseline all nodes, it was dropped due to resource availability, but all this discussion is making it clear that maybe it should be.
I voted for that one. The job that we could promote a bulk baseline to all the devices that had changed allowed up to keep the config changes for audits and ensured that our change reports were never too outrageous. The last one that came in was 7MB, the average before that was 680K. We have two reviewers and one person that set baseline. We are a small shop and having to go through the rigamarole to base line everything now is eating into other jobs. My manager asked me about finding another solution. I told him to wait a bit and see if the functionality is going to come back. I have been with Solarwinds a long time now, and hate to have to spin up another product just for this.
This is really great feedback, so thank you for speaking through your issues. I can see that perhaps we need to talk through this.
Would you be open in the future for discussions with myself or even our UX team? ashley.orr can put you on our list to help shape what we do with NCM going forward.
I want to agree with Thoar500 as well. We performed a process very near what he describes. We have 1500+ devices that we would do an weekly baseline report on and then baseline the entire network. That way we had an easy way to see if there were unauthorized config or config that was not cleaned up after a troubleshooting event. I'm definitely missing this function. I see no easy way to achieve a network wide baseline right now. Boo. 😞
Using baselines to keep an eye on changes to the configuration is still a working feature of the latest NCM version. Here is an example from my lab where I promoted one config to be the baseline. The system found and displayed it correctly that there is a mismatch in the config lines and hence my widget "baseline vs config conflicts" shows 100% conflict as I only have this 1 device.
Could you please check out this guide to creating baselines as it also lists the permissions required doing so: Establish baselines as a comparison point for config changes - SolarWinds Worldwide, LLC. Help and S...
Yes I agree with you that NCM 7.9 can still keep an eye on config changes but lets say you have 400 devices in your lab and you make a change to all 400 devices. Did you then have to manually promote 400 configs to baseline?
There used to be a job you could run to set as baseline in bulk.
Anthony - the optimized workflow in the feature now allows you to promote/create one baseline that applies to 400 nodes and not have to baseline 400 nodes with 400 individual baselines. Do any of the nodes you monitor have similar/same configs?
Sure I have groups of like devices...lets say I have two hundred routers that are similar in machine make/model and even similar in purpose (Branch Routers that connect to MPLS service provider.) But things like hostname, network statements, IP addresses, port descriptions, ACLs, etc, that make each config unique to that device and have everything to do with how that device functions, would all have to be ignored if we use a single baseline for all 200 of those devices.
And what if someone makes a change to an interface, or they inject a network into a routing protocol or they change some other line that I had to "ignore" in order to get that config to comply with the new baseline? I'd never know about it.
To me the feature you suggest we use is exactly what we are doing with Policy Violations Config Compliance Checker. So now I'm afraid that Baseline and Policy Violations Config Compliance Checker will both morph into one and we will be stuck retooling our Config Management process.
I agree with Thoar500, this functionality is crucial. I have almost 800 nodes, spread across multiple hardware types, vendors, and versions of code. Creating a baseline for each is a huge job that would need to be repeated any time changes are made. We have some devices, that there are changes on a daily basis, others that don't change except during quarterly maintenance. Staff is our most precious resource and we are now going to have to devote more time to this.
Maybe it is time to look for other solutions.
The scenario you described with the legacy baselining is an exceptionally rare situation. It appears to me that you guys actually built a process very closely around the capabilities of NCM, so it is obviously a pain if the tool changes. In the field I would say 98% of NCM owners just ignored the baseline feature completely because it wasn't well suited to what most people needed to do. Given that uptake was poor the NCM team seems to have tried to align the new baseline workflow with the kind of feedback that people were giving them, one being that the all or nothing baseline violation thing didn't make sense for the majority of them in environments with configs that are continuously moving forward. Nothing you described in your old workflow sounds impossible now, but the specific workflow you had and the specific reports you used will likely need to be reworked to align them with the new features. I do think you are correct in that there is a lot more overlap now between baselines and policy violations.
I'd say this is a good argument for testing out new releases in a lab/eval before deploying them to the prod instance, so you can see where changes might impact your existing workflows and you have time to plan out how you will handle the change. Probably a good idea to revisit the decision process for the old work flow, how much of that was actually driven by your requirements from the auditor and how much of it was driven by the way the tool worked at that time? I suspect you can probably cook up some new reports that give you the same level of visibility into changes as you had before, but it might take some time to work it all out.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.