Solarwinds NCM Bulk Password Change Best Practices

We are investigating Solarwinds NCM to perform bulk password changes for our network devices.

Can anyone direct me to a blog post, discussion, support article, etc. with best practices for this use case? Also, please let me know your detailed thoughts about using Solarwinds NCM Config Change Templates, Compliance (Report, Policy, & Rule) and/or Jobs for this use case. The Compliance (Report, Policy, & Rule - with remediation script) seems like the incorrect feature since rules are based on the presence of a string in the configuration which the trigger for a remediation script. However, a Config Change Template is triggered to execute by an user.

It may be helpful to share that I am somewhat new to using Solarwinds NCM. As a proof of concept, a couple of scripts using Config Change Templates have been created that seem to work fine on a couple of network device types. The desired result is to create a process for our operations team to be able to change the password for compliance reasons.

Find more posts tagged with

ncm_job

password_change

ncm compliance reports

cli

ncm

ncm config change template

ncm_scheduled_job

Accepted answers

All comments

sja

Take a look on that

https://m.youtube.com/watch?v=NOSPupDFsbg

That explain compliance...

YouTube is a friend..

mesverrum

If you are new to NCM the far easiest way to do this would be to just do a job with a script, since the CLI for changing passwords should be identical for the majority of your devices, and you wouldn't need to do any case by case logic so the CCT is just making a simple task into something more complicated.

To give you my typical use cases

Job running a script - Any time when the commands I'll be running are identical across multiple devices. I often will save "templates" in here of commands I run more often with just a few blocks I have to change before I run them and it feels like this is the fastest way to get changes through, but also leaves you somewhat more open to typos and other user errors.

Config Change Template - Times when I'll be running the same commands across multiple devices but I need to employ some kind of logic such as updating the configs of all interfaces with "uplink" in their description. To be honest this bit is a bit challenging for a lot of people but if you learn the syntax you can build some really impressive things. I see many people trying to use these because it minimizes the amount of CLI the users of the tool have to know. Essentially you can write the template such that it just prompts the user for the account name and password you want it to change and it would have all the logic built in to know which syntax to use, but getting to that level is somewhat rare as I meet lots of people who claim to have years of NCM experience who cannot write intelligent NCM scripts at all.

Compliance reports - When the change I'm making is something that I would consider to be a standard, as in making sure my trunks all contain the standard VLANs, community strings and authentication settings that should all be set up the same, ensuring that my ACL's are consistently defined where they need to be. You can have these as just an informational identified that something might not be right or if you really know what you are looking for and have the CLI commands handy you can set these reports up to identify and remediate config problems immediately. Also the obvious use as a tool for ensuring compliance with things like security policies/audits.

trek333

sja...Thanks for sharing, I have watched this video also. While I agree that the video is useful info, it does not cover the specific use case of bulk password changes. If you aware or become aware of content that covers the specific use case, please share with me and the readers of the post.

trek333

mesverrum...Thanks for your comments about NCM Jobs, Config Change Templates, and Compliance reports.

trek333

It may be that no list of best practices exist for using Solarwinds NCM to bulk change passwords. A list was started below. Please feel free to add constructive criticism and/or additional best practices.

Use current best practice to create a complex password that is difficult to decrypt. The technology/methods to decrypt passwords gets better every day.
The password should be obscured (use CLI commands that use one way hash/crypt for the password string) in the NCM script log file(s), all NCM logging disabled for password change scripts with plain text passwords or just disable the logging of the CLI commands with plain text passwords (is it possible?), and/or tight security controls for permissions to read Solarwinds NCM log file(s) by Solarwinds users.
- All of your network devices may not have a CLI command that has an one way hash/crypt for the password string. It is just a better practice to not have the password in plain text in any log file. Even as a hash/crypt text string, it provides some info about the password.
- Is anyone aware of all the NCM logging features and the steps to disable them for a specific script? A better feature would be to have special type of script variable that is a "password variable" which all NCM logging features would just output asterisks for its value and provide an input field that echoes only asterisks (see url link to feature request below). All/most major scripting languages have these features, but they were not found in the NCM documentation. And/or a feature to disable the logging for one CLI command (i.e. the password change CLI command). With one or both of these features, the script's log would be available for a compliance audit, but the actual password would not be in the log.
  - NCM Logging locations that I am aware of
    - Transfer status - show script results (.i.e. Solarwinds database)
    - NCM Advanced Settings (Enable Trace Settings - disabled by default & used for troubleshooting) - From Solarwinds UI - The session trace files can be found here: "%ALLUSERSPROFILE%\Application Data\SolarWinds\Logs\Orion\NCM\Session-Trace"
  - Related existing feature requests (including my own feature request)
- Tight controls on the Solarwinds user security is additional step to keep folks from viewing the NCM logs specifically for password change script executions.
A process to verify that the password was actually changed on the Solarwinds NCM node is advisable. For example, a script/CLI command that requires the new password to execute. Again, the use of the password needs to obscured if logged by NCM.

Based on the above feedback and my own investigation, a Config Change Template script may be the best choice within NCM since it can prompt for user input (list of devices, username, and password). Using a NCM Job script, the new password would need to be stored in a script or input file (assuming a NCM Job script can have a file as input). A NCM Compliance remediation script is triggered based on string in a network device's configuration. I am not aware of any configuration setting that would indicate a password for an user should be changed unless it is some sort of variable setting that is used with a customer's business process to indicate the "password life" has expired. Also, the new password would need to be stored in a script or input file (assuming a NCM Compliance remediation script can have a file as an input).

Rhetorical question: What is the ideal number of places to store a password? The answer is "as a few places as needed", and obscured in the places that are needed.