Customer issue:
NCM Nightly Config Job Stuck - Rebooted the system worked again and downloading the configuration .
Check List :
Check Windows Events : SolarWinds.net >> Errors
Check CPU Sockets:
Check the NCM BL / Job Logs:
Check CPU on the System :
Check System CPU is configured as recommended by Solarwinds
Check the NCM BL / Job Logs:
View live NCM Job Logs activity for any failure / Errors / Performance
Cause: NCM Job was running and it got disconnected with SQL Server for some reason
You can find the Logs under following location where you can check and track the NCM jobs activity if there is any Error there can be tracked.
C:\ProgramData\SolarWinds\Logs\Orion\NCM
NcmBusinessLayerPlugin
NCM.Collector.Jobs
NCM Nightly Job was running and for any reason (System Performance / L2/L3 / SQL Disconnection )
2019-11-16 04:00:11,716 [20] ERROR SolarWinds.NCM.Caching.CacheProcess - Timer tick errorSystem.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The network path was not found
NCM Unable to Convert the Global profiles as it failed to connect to the SQL Server
2019-11-16 02:00:10,946 [9] WARN SolarWinds.Cirrus.BusinessLayer.TransferExtensions - Unable to convert '${GlobalConfigTransferProtocol}' to CommandProtocol
Root cause: NCM Cache Service failed to close the NCM Job which was running
2019-11-16 02:00:10,962 [9] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error closing exception.System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Close()
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Close()
at SolarWinds.InformationService.Contract2.InfoServiceProxy.Close()
NCM Scheduler Tried to re run the NCM Job again
2019-11-19 02:00:16,596 [12] INFO SolarWinds.Cirrus.BusinessLayer.Jobs.JobScheduler - NCMJobScheduller: Found job to process- Nightly Config Backup
NCM Scheduler found the Job is already running Therefor Scheduler failed to re run the Job again
2019-11-19 02:00:16,596 [12] INFO SolarWinds.Cirrus.BusinessLayer.Jobs.JobScheduler - NCMJobScheduller: Job already running, updating next run time
Resolution:
So you do not have such issue in feature please follow the below post steps section
Consider System Performance / L2/L3 / SQL Disconnection
Troubleshooting NCM performance for jobs /devices downloading configs failure
NCM troubleshooting landing page
Still need assistance from Solarwinds Support
Tips and Tricks on opening a Support Ticket with SolarWinds
