This post will help you to troubleshoot most common issues with the NCM Jobs / Download Device configuration / Common Errors while downloading the Configuration / Running NCM jobs checking the NCM logs.
Please follow the steps and recommendation carefully and do let us know if this post helped to address your issue in comments section .
Please Note: Currently supported software versions Network Configuration Manager (NCM)
Latest Version: 8.0
Oldest Supported Version: 7.6
EOL VERSION | EOL ANNOUNCEMENT | EOE EFFECTIVE DATE | EOL EFFECTIVE DATE |
|---|
| 7.7 | June 6, 2019 | September 6, 2019 | September 6, 2020 |
| 7.6 | December 4, 2018 | March 4, 2019 | March 4, 2020 |
Solarwinds strongly recommend for an upgrade to the latest version with HF installed - More details you can find out under the posts below linked
Fresh Orion deployment Vs upgrade older version
Your check list
NCM version System Hardware & latest HF installed
Disable Session Trace
Clear Temp Folder
Clear pending reboot
Disable Config Archive
Increase CLI TimeOut
Reduce the amount of the config retain
Reduce the Simultaneous Download / Upload
Multiple APE ? - Create Job for each poller / One Job and one device from each poller
Possible a single or some nodes are culprit and failure download in the loop
1-NCM System &Hardware Sockets
Make sure your are on NCM 7.9 OR Upgraded to latest released NCM 8.0 + Latest HF installed version installed
Please audit your environment and make sure there is no bottleneck at your side which is causing the issue
How to check the server hardware using Orion Platform diagnostics
Open Task Manager On Orion Server > Performance Tab > CPU - Make sure you have Minimum 4 Sockets available there.
Make sure you are on recommended hardware
For more details please see My Thwack post below.
https://thwack.solarwinds.com/docs/DOC-190027
Check the HF from the Customer portal and make sure you have the latest HF installed
https://customerportal.solarwinds.com/HotFixes
2- Disable Session Trace
In few cases if you are running jobs for large network its not recommended to keep the Session Trace ON as it will consume CPU and Memory on the system also will effect the config download progress therefor keep this folder clear .
To disable Session Tracing:
- Open the Orion Web Console.
- Go to:
7.6 and older: Settings > All Settings > NCM Settings > Advanced Settings
7.7 and newer: Settings > All Settings > Product Specific Settings > CLI Settings - Clear the Enable Session Tracing check box.
- Go to the trace log location:
7.6 and older: C:\ProgramData\SolarWinds\Logs\Orion\NCM\Session-Trace
7.7 and newer: C:\ProgramData\SolarWinds\Logs\Orion\CLI\Session-Trace - Delete the trace files.
3-Clare Temp folder (Recommended If its larger )
Check disc space on NCM drives as well as SQL server where the NCM DB is stored
Clear the Windows Temp directory for all polling engines in NPM
- Log in to the Orion server hosting the Main Polling Engine.
- Stop all Orion services.
- Disable the Antivirus software running on the system.
- Navigate to:
C:\windows\Temp (including the SolarWinds folder) - Delete all files in the Temp folder.
- Restart all Orion services.
- Restart the system.
- Repeat step 1 through step 7 for all servers hosting an Additional Polling Engine (APE).
4- Reboot the NCM Server (Recommended ONLY IF there is windows update pending reboot or / AV could cause issues with NCM jobs)
5- Disable Config Archive
Settings > All Settings > Config Settings ( Disable Config Archive)
6- Increase CLI TimeOut
Go to the Settings > All Settings > CLI Settings > (Uncheck session Trace ) > increase the timeout values a bit .
7-Reduce the amount of the config retain in the NCM . (Recommended - Will improve the NCM & SQL Performance )
Off load extra load from the NCM DB will also help to run the NCM jobs faster.
Configs > Jobs > Edit the >Default Purge Configs Job
Follow the Wizard and on Add Job specific Details > Delete all Configs Except for the last 10 days.
For Newer Version please follow the KB
NCM Default Database and Archive Maintenance job
Most common issues with NCM Jobs area
Schedule Job is not running at all / stuck on 99% or 100% / Running 100% - Now Post Processing what should i try after above ?
I have a schedule job and it randomly failing for different nodes What to check ?
Turn On NCM Job Logs first
If its already enabled take a backup of the folder and delete all the old files files from the folder
Settings > NCM Settings> Advanced Settings > (Enable Scheduled Jobs)
You can also reduce the load from the poller CPU if its lower then 3.0 Ghz use the below settings .
Create a New NCM TEST Job > Add only One NODE (From main poller) and then run the job manually check if you have the failure ?
Now Check the same Job schedule after 10 minute?
What results you have ? Failure .
If you have success ad 10 more nodes into the same job and run it again and so on up to 50 nodes and then test up to 100 nodes in same job
It is Possible a single or some nodes are culprit and failure download in the loop this could cause delay completing the NCM job
Please make sure the Nodes you have added in the Jobs for testing NCM able to download the configuration manually without any long delays.
I have Multiple Polling Engines (APE)
If you have multiple polling engines Create separate job for each of APE and test the same as above
Create Job for each poller / One Job and one device from each poller
Please create Nighty Config Backup jobs with nodes only from one poller.
Please create this job for all pollers and run.
We need to know if there is a problem with a specific poller.
If issue is on a specific poller
Does manual config download work for this poller?
Could we test ping between Main and this problematic poller?
Maybe firewall rules block something. Could we do some tests - turn off the firewall and check if the job works? (could be problematic)
On the Orion Server go to the following location and check the log files .
C:\ProgramData\SolarWinds\Logs\Orion\NCM\Logging
Check the log file and see if you have any Error there ?
Open Support Ticket
Tips and Tricks on opening a Support Ticket with SolarWinds
NCM Inventory Job is taking to much time / Which nodes are taking long time / Where i can see real time inventory job logs
I have few nodes failing downloading config files (Connection Refused ) / (Connection TimeOut) Error what should i check?
Pick one single node and work along to make sure the node in question actually have no issues with connectivity.
(Please make sure you work on the correct polling Engine where the node is assigned for polling if you have multiple pollers)
Checking NCM Profile
Orion web console > Go to this node "target effected node" > Edit Node > NCM Properties check the Connection Profile "Test" if this successful ?
Checking NCM Profile with SSH Auto
Also please on Orion web console > Go to this node "target effected node" > Edit Node > NCM Properties check the Connection Profile
Select SSHAuto >
"Test" if this successful ?
If you have connection failure (Make sure you RDP on the the same Poller where the node is assigned )
Please try this "ConnectionTester.exe" tool from NCM server and let me know the outcome if this failed as well ?
C:\Program Files (x86)\SolarWinds\Orion\NCM\Tools
ConnectionTester.exe
If you are able to connect to the node without any issue and still have the same issue in NCM downloading the configuration
Or
You are able to connect with the device using PUTTY / SSH or NCM Connection Tester however you have failure with NCM when running Connection Profile "Test"
In this case we have to check the Session Trace for the Node.
Enable Session Trace
- Open the Orion Web Console.
- Go to:
7.6 and older: Settings > All Settings > NCM Settings > Advanced Settings
7.7 and newer: Settings > All Settings > Product Specific Settings > CLI Settings - Enable Session Tracing check box.
- Go to the trace log location:
7.6 and older: C:\ProgramData\SolarWinds\Logs\Orion\NCM\Session-Trace
7.7 and newer: C:\ProgramData\SolarWinds\Logs\Orion\CLI\Session-Trace - Delete the trace files.
Now Run the below Test and check the Session Trace log file for the Error (Error will be listed on the bottom of the log file)
Checking NCM Profile
Orion web console > Go to this node "target effected node" > Edit Node > NCM Properties check the Connection Profile "Test"
Checking NCM Profile with SSH Auto
Also please on Orion web console > Go to this node "target effected node" > Edit Node > NCM Properties check the Connection Profile
Select SSHAuto >
"Test"
If you are unable to understand the Error why its failing - You can either search or post the issue on the Thwack or Open Support Ticket and provide us the Session Trace file.
Please do not forget to ZIP the Session Trace file
Open Support Ticket
Tips and Tricks on opening a Support Ticket with SolarWinds
NCM nodes failing downloading Error message: connectivity issues, discarding configuration / "show running" on a Cisco switch ( % Invalid input detected at '^' marker. )
Troubleshooting downloading F5 devices configuration
NCM troubleshooting downloading F5 devices configuration
NCM Logs and data locations
Where i can see NCM Jobs activity in details ?
You can find the Logs under following location where you can check and track the NCM jobs activity if there is any Error there can be tracked.
C:\ProgramData\SolarWinds\Logs\Orion\NCM
NcmBusinessLayerPlugin
NCM.Collector.Jobs
Default location for CLI and Session Trace Logs
C:\ProgramData\SolarWinds\Logs\Orion\CLI\Session-Trace
C:\ProgramData\SolarWinds\Logs\Orion\CLI
Default Location for NCM ASA Polling
C:\ProgramData\SolarWinds\Logs\Orion\ASA
Default location for NCM vulnerability location
C:\ProgramData\SolarWinds\NCM\Vuln
Default location for config archive
C:\ProgramData\SolarWinds\NCM\Config-Archive
I will include more details in it and case studies please feel free to let me know about your feedback and i will include in this guide.
Related Link.
NCM troubleshooting landing page
