NPM Polling stops. No Rhyme or reason.

We were running NPM 9.0. Polling would stop at random occasions. The service itself was still running, but no informaiton was being updated on the webpages.

Stopping and starting the service did not resolve the issue. Rebooting the server and all would be back to normal besides the missing info from the time frame the polling stopped.

Seeing that 9.5 was out. I went ahead and updated with a few hitches that SolarWinds support helped out with. We are still having the polling stop at random times. Could be middle of the weekend or middle of the day... or late at night. Service is still running. Stopping and starting the service does not clear up the problem. Rebooting the server does.

I have made a ticket (Case #99603) but wanted to post to see if others have seen similar issues. I did read other posts but they seemed to be some time in the past, and they had reported stopping and starting the service resolved the issue.

Thanks for your time and assistance.

Find more posts tagged with

orion_npm_9.5

network_performance_monitor

Performance

lsass

apm

64-bit

netperfmon

poller_stop

orion_npm

windows_2008

Accepted answers

warbird

Now, the point is if we can do something else until we can get the phisycal server assigned?

Reduce your statistical polling interval. If you are running NTA, reduce the number of flows pointed at your primary poller.

I mistyped. To clarify what I meant: You can decrease load on the server by slowing down how often the statistical polling is done. You do this by increasing the time between polls (increase the interval, not decrease as I mistyped).

All comments

pg01

our server had this problem sometime back, v9. The NetPerfMon would stop polling. A stop / start on NetPerfMon did the trick. We haven't had this problem for past several weeks. One thing you can do is setup a tcpdump on a separate server, for ICMP, and then email if its stops seeing pings.

bshopp

Sounds like you already have a support ticket, might also want to go to latest and see if anything changes, we released 9.5 SP1

tomciscoalbert

I was having the same problem with 9.1 SP2, it was an 4 GIG MEM/Quad Core Zeon SLX polling engine, 8 GIG MEM/DUAL QUAD core SLX NPM and a 16GIG MEM/DUAL QUAD CORE XEON SQL cluster, I had a case open for a few weeks so after multiple options suggested by SW support there was no changes and no idea why it was stopping, when 9.5 came out and since the server was not in production at the time I upgraded it to 9.5 and so far it has not reocurred. having been a Network Engineer for 10 years and resisting Cisco's first response to upgrade the code as soon as I open a ticket, I would not suggest this for production enviroements unless you can justify it with out first testing it in non production enviroments.

mcampbell

Much appreicate all the responses.

I applied the SP 1 for 9.5 on the day I posted the above message (after I posted that is). That evening the polling stopped.

Since the last halt, there has not been another one. The web interface is moving at a better pace, but no where near the speeds it was running at when we were running 9.0.

We will sit and wait to see if the problem has actualy gone away. But I am eager to see some more improvements to the web interface's speed.

Thanks.

bshopp

If it keeps happening, be sure to open a support ticket. We have made some more improvements in 9.5 and continue to do so with each and every release

looperz

We have a similar problem when polling has stopped everynight after we upgraded to 9.5 SP1. It looks to be around the same the the DB maintaince runs. We are currently running the maintaince now during the day and it looks like polling has come to a halt and the DB server is maxed out.

horric29

We have the same issue... Been working on it for two weeks... Just randomly stops and there are never any logs or reasons for the support group. Today I found some patches for Server 2008, including a few SNMP fixes for mass SNMP polls from the OS that would cause the SNMP engine to fail, that didn't work, then found there were more SNMP fixes in SP2 for 2008. I've gone to that and have been stable for a few hours now... I'll give an update tommorow.

mattinglyd

I was using SNMPv3 and this seemed to be the cause. This was occurring on random devices - if I moved them to SNMPv2 the problem would stop.

DGoodale

I had nothing but problems with SNMPv3. After trying to struggle through it with my Orion and CiscoWorks servers, I finally gave up after 3 months of grief. SNMPv3 is much more secure than SNMPv2 but requires extensive configuration changes on the devices themselves to allow access to the data in the MIB tables. I had a few devices that just would NOT talk SNMPv3 no matter what I did. What a relief to roll back to v2.

Dan Goodale - Network Engineer, Triwest Healthcare

mcampbell

I am the OP. This problem is still occuring. At present, we have worked with various people at SolarWinds. The original ticket got closed for resolution on some corrupt database issues.

They started a new ticket to log fresh working on this issue. Case # 115898.

I have been in contact with Sr Tech management, as well as developer. They have me running a debug dll for the issue and uploading a diagnostics after each failure.

They are hinting at it possibly being caused by a specific device I am polling, but I personalty am not feeling strong in that direction. I am defering to their knowledge of course as it is their product.

Presently I have Senior Management here at my company very frustrated at the cash we spend on this product, with how long this issue has been going on.

byrona

Thanks for your reply though it makes me feel even worse about the issue than I already was as it sounds like a much bigger issue than I had originally thought.

We also just sunk a bunch of money into this product for a very large project that we are working on. We are pretty much supposed to be going live with this project this week for the customer and if this problem doesn't get resolved Sr. Management is going to be very upset at the money they just spent, especially considering the product we were using before was Open Source and pretty much not costing us anything.

shorn

I work with byrona (above) and am also trying to get this resolved. Since it seems like SolarWinds is aware of the issue I'm going to focus on detecting the problem. Since the engine doesn't actually stop, detecting the problem is a bit tricky. Does anyone know of a way to access the poller statistics from a script? We have noticed that the number of SNMP Polls Outstanding counter on the Engine Status web page starts climbing when this occurs. My thought is to create a script that checks this value periodically and then sends an email when it crosses a threshold, I'm just not sure where I can grab the data I need. Does anyone have any suggestions? My preference would be to just access some of there application components directly instead of parsing a web page.

MarieB

Hello Everyone--

EEK! Lots of posts on this. Just want to let you all know that I've contacted the Product Managers about your concerns and I know they'll get back to you asapy to get this addressed.

Thanks for your patience.

Marie

bshopp

Hey all, we are aware of this issue and are actively working it to further diagnose and see what is going on. The Dev team has been working directly with some experiencing this issue to further investigate. Do each of you have support cases open? If so, could you PM me via thwack the case numbers please?

swack

I've e-mailed a request to the engineer working on my case, asking that it be escalated up the chain. I also included a link to this thread so they can maybe start seeing the pattern of similar issues from multiple customers.

byrona

A few more items I thought I would bring up as we go through this problem...

1) Would AntiVirus software on the polling engine and/or the database contribute to or cause this problem?

2) Which method of SQL DB redundancy would be best for Orion DB? We are currently using log shipping but are not sure if that is the best in this situation.

rdollins

I am running SNMPV3. At any time a node will no longer be able to get stats on interfaces and volumes. Stats from Universal Device Poller on the node keep working with absoutley no problem. If I have the AIX crew restart the SNMP daemon, all will return to normal at once for the interfaces and volumes. But this issue occurs on several differnt types of devices so it is not limited to AIX devices.

I start running a packet capture on one of the devices that are not able to get interface stats.. What I see is the the poller is sending the correct engine boots and correct engine time for everything except the interface stats. He sends an engine ID of 0 and engine time of 0 when he asks for interface stats, he gets a reply back that says "NotInTimeWindows" or "unknowEngineID" .Then a few seconds later when he is polling for a Universal Device Poller stat he goes back to the correct Engine boots and correct engine time, and he of course gets back a good reply..

This behaviour can start on any node at any time.. The ugly fix is to restart snmp on the device or delete the node from Orion and recreate, and you are good.. Any idea on what is causing this to occur?

Rick

FormerMember

I am also experiencing random polling stops. I notice it is always my primary poller and it happens most often over the weekend. I usually kill the still running netperfmon.exe task, restart it, start system manager and connect to polling engine. It takes a while but these steps generally bring back the poller. The poller only stops in the early morning (3~5AM) hours. Every time I reboot the server (2003) running the poller I get messages about sending dumps to microsoft. These dumps are produced by the business layer process. I'm running 9.5 w/current patches. We have decided to NOT buy APM and NPM because of this unstability. Since I run an unlimited license, this is costing SW a bit of revenue. Support's been good but the problem is not fixed.

BryanBecker

Add me to the list...I had my 1st occurrence last week on my new 9.5 SP4 server. Since I was out sick most of the week no one noticed until like 4 days later. I have 2 new 9.5 SP4 servers connecting to the same SQL server. All the servers are on the same LAN basically yet only 1 of the 2 had an issue. There has to be a way to add a service checker that if the netperfmon service is stopped to try and restart it....if it continues to fail send an email out or something. I've seen this many times in various Orion versions that we could lose days of data if someone isn't constantly checking.

Now in Windows Services all of these are set to "Restart the Service" on the Recovery tab but that doesn't seem to be the case.

byrona

Well, there is a bit of hope. SolarWinds got us a development patch yesterday and the very early preliminary results look good. According to our support engineer our diagnostic files indicated an issue related to wrong handling of some different behaviors of the WinSock API under Windows 2008.

I will keep you all posted!

warbird

I have been experiencing a similar issue but not positive it is exactly the same. It seems that when we add new nodes or interfaces to our secondary poller, it will crash usually within 24 hours. 9.5 SP4, 32 bit Windows, 2 pollers, standalone SQL server, etc.

The netperfmon service will eventually crash, after having gone sideways for many hours, which is noted by the fact it would start doing db syncs at larger intervals until finally crashing altogether.

Also, the netperfmon service would not auto recover, even though we have it set to in Windows Services. It has to be manually restarted.

I have not opened my own ticket on this yet, as I have been trying to evaluate the cause.

TimothyGaray

I'll weigh in for our similar problems.

Running 1 Orion NPM, APM, NTA and NCM server and 1 polling engine only. Both are VMware virtual machines. Both are Windows Server 2003 Standard x64.

Same situation, polling stops. Trying to restart service is futile. Rebooting gets things running.

Did anyone find how to monitor and thereby alert on the SNMP Outstanding value?

Thanks!

-Tim

FormerMember

Hi, I put this message in yesterday, forgot about this trail. Check your SQL server and Orion logs carefully:

Most of you probably know this but just in case:

Symptom, Poller seems to stop at a certian time each week/day etc. My root cause was the DB group cycles SQL on all servers every Sunday morning to clear various SQL memory issues etc. Not a re-boot, just a down and up on SQL. Anyway, every Monday I'd see red on the monitoring polling engine. I spoke with the SQL team and they do it every Sunday at 3:00 AM. I created two scheduled tasks on mu Orion poller to pause netperfmonservice at 2:55 AM Sunday and to continue at 03:15. Works nice and clean, poller is up all the time now.

syntax at the command prompt on the server:

at 02:55 /every:Su net pause netperfmonservice

at 03:15 /every:Su net continue netperfmonservice

Lots of different ways to accomplish this looking for SQL messages, trap functions etc. This was simple and works good enuf.

Thanks

savell

We use the following SQL statement as an ADO user experience monitor within ipMonitor.

select ServerName, KeepAlive, GetDate(), DateDiff(second, KeepAlive, GetDate()) from [dbo].[Engines] where DateDiff(second, KeepAlive, GetDate()) > 60

If the row count returned is > 0, then at least one of your pollers hasn't updated the database within 60 seconds (indicating an issue with the polling process).

ipMonitor then sends out e-mails warning of the failure. The same SQL could be used in a number of different forms outside ipMonitor (i.e. scheduled SSIS job for instance).

Dave.

Edit: I should note that we did this as a result if seeing this issue within our implementation. In our case it was the result of a switch failure (reboot) that occurred from time to time - which provided the network connection to a couple of pollers and the database server. Once the pollers got behind, they seemed to never recover without a restart. The switch was replaced and the problem has not re-occurred. A similar problem I imagine would occur with a database restart, firewall connection issues etc.

firehawk_350

byrona, how is the dev patch working? We currently have this problem with a 5 NPM pollers, 2 APM pollers, and 2 Netflow collectors.. I see a ton of "time out for buffer latch wait" in addition my DB will start to freeze and SQL is only using 8gb of the 24gb on the dedicated SQL server. I look in the DB and there is a ton of blocking going on inside netperfmon db. We are running on Windows2008 64 bit.

byrona

The patch seemed to pretty much solve the initial problem and I think (though I am not certain) that the fix was rolled out in the post recent set of patches for everybody.

Now we think we may have a different polling problem where some node level collections will stop on groups of systems at a time. This problem is a bit more elusive as it doesn't seem to happen frequently and only on some things when it does. We are working to better understand this problem.

If you haven't already you should try applying the most recent SW patches and see if that helps.

warbird

byrona,

Do you notice this happening after you have added new nodes or made a bunch of changes (edited node names, added interfaces on already monitored nodes, etc)? I have been noticing that my secondary polling engine will intermittently go sideways after many new nodes are added or many existing nodes have been edited. "Many" is a loose term here, as it does not happen if only 2 or 3 nodes are added/edited. The intermittent nature is making it difficult to track down or collect data.

BryanBecker

I'm in the same boat....services up but no data is collected. I'm even seeing that launching System Manager shows no devices. I have to stop all the services and restart them and it works again.

If there is a beta dll or a fix I hope they get this out ASAP! I have seen 2 scenarios where this needs to be fixed. 1. The services are running but no data and 2. The services just stop for no reason.

Many people, including me, have asked for a way to trigger an alert on these issues and not find out days later of the issue.

bshopp

Bryan, do you have a support case open about this?

mondtw

We are also having the issue where the polling engine stops nightly and we need to restart NPM service to get it running again. I've had a ticket open without resolution. They have indicated that our poller is over-loaded but with no changes in elements, polling intervals, or polls per second a system that worked flawlessly under 9.5 now fails nightly under 9.5.1.

Lertsa

Hi all,

there's another thread of a problem that has similar symptoms than this one, not sure if everyone is experiencing them the same way. ()

Figured I should share my views and experiences of this problem on this thread too if it would help someone to figure out why this is happening. Bolded some points that might be of some use.

I've had this problem (Orion NPM stops polling without any visible reason) with versions 9.5.0 SP4 and 9.5.1. Got some help from the Solarwinds helpdesk and we checked the databases for mangled IP's, duplicates etc. Also disabled UAC and put Orion running on a local administrator account instead of the system account. I also renamed all my nodes so that no underscore was used (there was something about this in the manual somewhere...) After these changes the Orion stayed up&running for 2,5 weeks without problems until now. Earlier these changes the problem appeared usually within a week. NPM service restart fixes the problem.

Earlier, every time this happened the server event viewer application log was flooded with "NetPerfMon Service Error: Exception in Service Timer Tick - 0-error". When this situation was on, I couldn't start the System Manager because of the compatibility error. The flooding starts with 5 seconds of 1850 entries and after the 5s, one entry every 5 seconds.

Now the same things happen but I've noticed two separate occasions where NPM stops polling but there's no flooding in the application log. Yesterday morning the polling had stopped at 7.15 but the application log flooding started at 9.51 and at the same second I logged on the server using domain admin account. There were no log entries in between.

If this is indeed a platform/version problem, ppl should compare their environments to see if there's something to consider.

Our environment:

Software:
Orion NPM SLX versions 9.5.0 SP4 and 9.5.1 tested
381 nodes 4800 interfaces

Operating system:
64-bit Windows 2008 server SP2

Database:
64-bit Microsoft SQL Server Express 2008
Located on the same server with the software

Hardware:
Intel Xeon E5540 @ 2,53GHz, 6,00GB RAM

Antivirus+firewall:
F-secure antivirus for servers 8 (upgraded now to 9)
Can't remember if the Windows firewall was on or off during the testing, F-sec installation turned it automatically on.

Others:
The server is connected to the domain, the Orion services are running on a local administrator account.
.NET 3.5 SP1 with update and hotfixes

Hope someone figures out a fix for this soon, I have been struggling with this for almost 6 months now (the server motherboard was defected and that took some weeks at first...). The old EE-server was shut down already so a working NPM is really needed to monitor the network.

- Lertsa

ps. I'm opening a new ticket when the problem appears again.