NetPerfMonService hangs for 15+ minutes after reboot (version 8.5.1)

Yesterday we upgraded Orion from version 8.1 to 8.5.1. After doing so, we are now seeing the following:

For approximately 15-20 minutes after the SolarWinds server is rebooted, the NetPerfMonService does not completely load. During this time period, I am unable to load System Manager - Network Performance Monitor (it hangs on the splash screen) and more importantly, my polling engine is not running (thus, no alerting but test alerts do work when kicked off in Advanced Alert Manager).

I'm getting around this issue by letting the server sit there for 20ish minutes and watching the Windows Application log closely after a reboot (the NetPerfMonService is logged as started 20ish minutes after the reboot, even though it shows as "started" in the services). I can also open the Monitor Polling Engines tool and see that my Polling Engine is not running. Once I see the service startup in my Application log, I see the polling engine change to "running" (after a refresh). I am also able to open System Manager - NPM. Actually, if I open System Manager - NPM at any point before the service has started (hanging splash screen) and wait for the service to start, it continues with the opening of the program.

Unfortunately, it's unacceptable to us that we aren't polling and therefore aren't receiving alerts during this 20 minute window after a reboot. I've logged a ticket with SolarWinds on this issue, but it would be interesting to hear if anyone else is seeing anything similar.

Let me know if you'd like any clarification or if anyone has any ideas/thoughts.

Thanks

Find more posts tagged with

v8.5

Polling

8.5

orion_network_performance_monitor_polling

Accepted answers

All comments

Malvado

Check the windows event log and see if you SQL Server is doing a recovery after the reboot, I also had this issue. If SQL is doing a recovery on the database nothing will connect to the database or start up properly until the database comes online.

FormerMember

Where would I see that? System log? Application log?

Nothing is jumping out at me, but I'll keep digging. What did you do to resolve the issue?

Thanks!

Malvado

It would be in the application log if sql is doing that, they come up as informational messages so nothing stands out.

It happens if the database is not clean when it was shut down, if something is still trying to write to it or the checkpoints have failed, etc.... still happens every now and again, I just make sure I have all services stopped before I reboot the server.

ravirsc

I also have this issue and Solarwinds support told me that it is due to high number of polling elements (9000+) and we should have another polling engine.

Are you also have high number of polling elements? (Above 8000+)

DeanBennett

I'm seeing similar symptoms with Orion v9.5 SP2 with APM (unsure of version, can't readily find it).

Sometimes when I reboot our Orion server (following routine maintenance work for example) I get a series of errors in the Event Log as follows :

SolarWinds.Net Event Log

Event ID		Source
4001		SWService
5015		SyslogService, SWService,AlertingEngine
1004		SyslogService

Application Event Log

Event ID		Source
18456		MSSQL

My APM license summary shows I'm monitoring the following :

Allowed Number of Component Monitors:		unlimited
Total Number of Component Monitors:		484
	Licensed Component Monitors:	484
	Unlicensed Component Monitors:	0
Available Component Monitors:		unlimited

NPM License details show the following stats :

Network Elements	1137
Nodes	204
Interfaces	706
Volumes	227

This issue is frustrating as we rely heavily on Orion during maintenance work to establish the overall status of our systems and as our primary tool for assessing status is not coping with being rebooted, it makes it very difficult to build confidence.

Any advice on this would be gratefully received.

bpmullen

Hi all,

I recently had the exact same issue. I am running 3 servers (sql, Poller and webserver) on a vm platform and discovered after using the "gaps"diagnostics that the hard disk writing was being cached on my sql server. I reconfigured the disk usage shares for the vmware so that the sql server had priority over everything else and the performance has increased dramatically.

The poller server used to take about 20 minutes or longer to boot up and log into system manager, which now takes about 2 minutes on first open. Polling stats are also much better with no gaps in the graphs. Here's the link for the diagnostics :

www.solarwinds.com/.../gaps.htm

ravirsc

I am not facing this issue any more with Latest version 10.1.2. So it got resolved now.
Thanks,

_Ravi_