Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Faulty polling engine - again

I have a setup with dual polling engines. Suddenly, for no apparent reason, one of them might fail to collect data, leaving gaps in the graphs (one dot every hour or so). Normal interval is 10 minutes. I have been through this with support, and they really don't have a clue. I have done all upgrades, I have an external SQL-server and the polling engines have been installed on both quad core blade servers, and now back on normal 3GHz/2GB stand-alone servers.

Restarting the polling engine results in normal behaviour. However, the process takes a long time to shut down, resulting in the error "Can't start service" if I use the restart option. But stopping (and waiting) and starting works.

This is actually quite a severe flaw, because I can loose a lot of statistics just because I didn't check on the polling engine.

Now - since this seems to be an unsolvable issue(?) - are there any ways of monitoring this and triggering an alert of some kind?

xref I don't know how many support tickets I have opened on this issue. SW should have all data and everything they'd possibly want to be able to resolve this.

Find more posts tagged with

Accepted answers

All comments

FormerMember

We experienced the same problem too. You say you have dual polling engines. The redundant polling engine comes with a copy of System Manager. You should not use the redundant poller System Manager to add/delete/update Nodes/Interfaces/Volumes. We discovered if both SMs were in use at the same time to update the database one or both of the pollers would "hang" however, the NetPerfMonService would still be running, SNMP polling would stop and eventually after about 3 hours the service would stop (Windows would not automatically restart the service either). Support recommended only using the System Manager on the primary poller (which hosts the Orion website). They said they would address this in a future release so I don't know if they have fixed this in 8.5 or upcoming 8.5.1. We disabled the .exe on the redundant poller and instructed our NOC to only use the primary poller SM and CPE (Custom Property Editor) to update nodes/interfaces/volumes. I'd say we've had almost a 99% improvement since doing this. Every once in a while we have a recurring incident and have to restart the service. We've been unable to isolate this particular case at this time. Also, I believe upgrading from 7.8.5 to 8.1 improved the situation as well.

What we have done to warn us of the pending doom of the poller is to implement an Advanced Alert to monitor the server adapter traffic. In our case, our pollers fluctuate around 400+ pps. When the poller would go into the "hang" condition, the pps would drop to almost zero. So we set the alert to trigger when pps < 100 (reset > 300 pps) and sustained for > 20 minutes (2 SNMP polling intervals). The alert triggers a page to Orion SysAdmins and they stop (kill, "end process") and restart service on the respective poller. Like I said we seldom have this occuring any more since we isolated the issue to running both SMs to update the database.

We also have acquired the Hot-Standby feature and it kicks in when one of the pollers "hangs" or stops now only lose stats for only one or two polling cycles.

Hope this helps.

Network_Guru

Is there anything written to the Windows EventViewer when the polling dies?
If so, you could use an app like Snare to send a syslog to another server & then email/sms the alert, until you can get this resolved.

Do you have a DBA that can analyze you DB when this is happening?
Perhaps some tables are getting locked by another process or ???

I'm polling over 6000 elements with a 3 year old dual core 2.3Ghz, 3Gb RAM server running SQL2000 DB & it is slow, but workable.
My stats polling is not aggressive (15 minutes for interfaces & 5 minutes for ICMP).

btp

Thanks for the advice.

I just timed the web-site, and it takes 28 seconds (!) to display the page with all the nodes. Also NPM has gotten considerably slower after the latest releases. I am polling ~5.500 elements (nodes and interfaces).

I really feel that if this progress doesn't stop very shortly we have to look elsewhere for a useful product. Right now, this is not.

FormerMember

We know of couple of resources that have issues with a large number of nodes. You might try detaching each resource individually (by clicking on the name of the resource) and seeing which ones load slowly. That would help us figure it out, and I can tell you if those are fixed in 8.5.1 or if we need to look at others.

btp

Here are the resources I have on my front page:

Managed nodes that are currently Offline (structured by 3 custom properties)

Search Nodes

Interfaces on Juniper router, 8 highest in traffic

Last 25 syslog messages of a certain kind

Down interfaces on all HP switches

Last 30 events

Last 50 syslog messages

I found something fishy though; the lag in login seems to be caused when going from Internet through my firewall. If I login within my management network, it takes 2 secs. Seems that I want to search elsewhere for the root cause here.