cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

NCM Nightly Jobs suddenly stuck - Reboot System worked [Case Study]

Customer issue:

NCM Nightly Config Job Stuck - Rebooted the system worked again and downloading the configuration .

Check List :

Check Windows Events : SolarWinds.net >> Errors

Check CPU Sockets:

Check the NCM BL / Job Logs:

pastedImage_6.png

Check CPU on the System :

Check System CPU is configured as recommended by Solarwinds

pastedImage_9.png

Check the NCM BL / Job Logs:

View live NCM Job Logs activity for any failure / Errors / Performance

Cause: NCM Job was running and it got disconnected with SQL Server for some reason

You can find the Logs under following location where you can check and track the NCM jobs activity if there is any Error there can be tracked.

C:\ProgramData\SolarWinds\Logs\Orion\NCM

NcmBusinessLayerPlugin

NCM.Collector.Jobs

NCM Nightly Job was running and for any reason (System Performance / L2/L3 / SQL Disconnection )

2019-11-16 04:00:11,716 [20] ERROR SolarWinds.NCM.Caching.CacheProcess - Timer tick errorSystem.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The network path was not found

NCM Unable to Convert the Global profiles as it failed to connect to the SQL Server

2019-11-16 02:00:10,946 [9] WARN  SolarWinds.Cirrus.BusinessLayer.TransferExtensions - Unable to convert '${GlobalConfigTransferProtocol}' to CommandProtocol

Root cause: NCM Cache Service failed to close the NCM Job which was running

2019-11-16 02:00:10,962 [9] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error closing exception.System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace:

   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)

   at System.ServiceModel.Channels.CommunicationObject.Close()

Exception rethrown at [0]:

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

   at System.ServiceModel.ICommunicationObject.Close()

   at SolarWinds.InformationService.Contract2.InfoServiceProxy.Close()

NCM Scheduler Tried to re run the NCM Job again

2019-11-19 02:00:16,596 [12] INFO  SolarWinds.Cirrus.BusinessLayer.Jobs.JobScheduler - NCMJobScheduller: Found job to process- Nightly Config Backup

NCM Scheduler found the Job is already running Therefor Scheduler failed to re run the Job again

2019-11-19 02:00:16,596 [12] INFO  SolarWinds.Cirrus.BusinessLayer.Jobs.JobScheduler - NCMJobScheduller: Job already running, updating next run time

Resolution:

So you do not have such issue in feature please follow the below post steps section

Consider System Performance / L2/L3 / SQL Disconnection

Troubleshooting NCM performance for jobs /devices downloading configs failure

NCM troubleshooting landing page

Still need assistance from Solarwinds Support

Tips and Tricks on opening a Support Ticket with SolarWinds

Version history
Revision #:
1 of 1
Last update:
‎11-21-2019 03:55 AM
Updated by: