after upgrade to 10.1 - getting tons of missed polls and general bad performance

Hi Forum and specifically solarwinds.

I upgraded to 10.1 last week. The upgrade was the usual a complete nightmare, but we are up and running now. I got the flood of “node rebooted” alerts as expected, but what I have found since the upgrade, is that I get tons of what appear to be missed Polls.

ALl nodes run:

NTA, APM, IPSLA

Node 4 runs NCM

My evidence is a little circumstantial, but valid all the same. I have an alert that tells me when any node looses SNMP pollability (is that a real word?). So basically, it looks like this:

Trigger:

Trigger when all of the following apply

Node status is not equal to down

Trigger when ANY of the following apply

Volume responding is equal to N

Volume status is equal to Unknown

Volume Percent used is empty

Trigger when condition has existed for 15 minutes

This sends me an email to tell me the device is no longer returning SNMP. I jump onto the box and restart SNMP, or rescan. This is to make sure I pick up on when someone rebuilds a box, replaces hardware etc. it works a treat….

I have a similar alert for interfaces.

Since the upgrade, these have been flooding in around 500% more often than before. Previously, they were generally true, i.e. the device really did have an issue, such as it had hung or it was a device configuration issue. However, since the upgrade, these are 80% false. The server has missed an SNMP poll, but PING has returned. By the time I get to look at the alert, the node is back to normal.

Disk queues on the SQL server are very low

You might ask what my poller statuses are, so here they are, about the same as they were before the upgrade, so it is a little worrying:

Polling Engine Status

Last Database Update Now

Polling Engine on SERVER1

Last Database Sync 1 second ago

Network Elements 217 Nodes, 658 Interfaces, 274 Volumes, 1149 Total Elements

Polling Completion 99.66%

Operating System Windows 2003 Standard Edition

Service Pack 2.0

Package

Polling Engine on SERVER2

Last Database Sync 2 seconds ago

Network Elements 302 Nodes, 459 Interfaces, 982 Volumes, 1743 Total Elements

Polling Completion 99.72%

Operating System Windows 2003 Server

Service Pack Service Pack 2

Package

Polling Engine on SERVER3

Last Database Sync 1 second ago

Network Elements 390 Nodes, 994 Interfaces, 403 Volumes, 1787 Total Elements

Polling Completion 99.67%

Operating System Windows 2003 Server

Service Pack Service Pack 2

Package

Polling Engine on SERVER4

Last Database Sync 1 second ago

Network Elements 645 Nodes, 1931 Interfaces, 10 Volumes, 2586 Total Elements

Polling Completion 99.70%

Operating System Windows 2003 Server

Service Pack Service Pack 2

Package

Polling Engine on SERVER5

Last Database Sync 3 seconds ago

Network Elements 758 Nodes, 1818 Interfaces, 77 Volumes, 2653 Total Elements

Polling Completion 98.83% - this one has said 98% even when it has zero nodes on it, so not sure I trust any of this information

Operating System Windows 2003 Server

Service Pack Service Pack 2

Package

Polling Engine on SERVER6

Last Database Sync 1 second ago

Network Elements 46 Nodes, 25 Interfaces, 94 Volumes, 165 Total Elements

Polling Completion 99.99%

Operating System Windows 2003 Standard Edition

Service Pack 2.0

Package

Find more posts tagged with

10.1_upgrade_polling_performance

Accepted answers

bshopp

All comments

lhorstma

Couple of things, first, sweet alert for the volumes not responding to SNMP, I'm stealing that one. I just have a report I go through every so often to fix the chronic ones. Anyways, I've got a few cases open with support that sound similar so I'll relate my issues here in case it helps... The first one was for resource utilization on the pollers, specifically CPU (but memory usage has spiked too). See below for my primary pollers CPU graph from the last 30 days. I'll leave it as an exercise for the reader to figure out which day we upgraded, with bonus points if you can point out reboots, service restarts, and the day we added 2 more CPU's:

Next, for missed polls, we have had several events where volumes stop recording data altogether, they appear fine (don't have the ? icon), they respond to a list resources, and deleting the volumes and re-adding them does not work to get them recording data again (only list as unknown with no last database update time/date). Restarting Orion services on the polling server is the only way to get them to start talking again. Of course we only figured this out AFTER the holiday weekend, so I have some volumes that now have a big multiday chunk of missing data that I have to try to explain to server owners. Orion services are all up, polling completion % stays above 99.5%, SNMP Statistic Polling Index stays pretty much at 7900 out of 7900, and the SQL server avg disk queue length stays below 1 for 99% of the time with only the occasional spike. Polls per second tuning has been tweaked to suggested settings on all pollers. Haven't correlated it to any specific events yet (though there have been various service restarts at other times that MAY be related, like the collector data processor fails every time ntbackup backs up the c: drive and system state).

This is the C: volume of poller3, which is monitored by poller2. During the event looking at the volume polling details will often list the next poll and next rediscovery as times in the past. Other than the more major events there are also regular missed polls and late polls, my statistics polling interval is 10 minutes, this morning for the above volume I have had polls at 8:37, 8:48, 8:59, 9:10, 9:22, 9:37, 9:49, 10:00, 10:24, 10:35, and 10:46. Mostly every 11 or 12 minutes with a missed poll around 10:11.

bwicks

Have to say I was worried about performance gains after upgrading as well. lhorstma curious how many interfaces you have and what is the Status polling interval? I posed the performance question while in RC to Brandon and he confirmed a slight uptick in CPU and memory utilization but your graph has a huge increase in it.

FormerMember

Hi there

Good reply dude, lots of detail. I just did the same as you and got some stats from a couple of my pollers. I can see a 10% increase in CPU utilisation on pollers since upgrade, and around 10% on memory utilisation too

I have not profiled the disk queues ont he database, but i wonder is there anything new hammering the DB?

this is madness, can someone in solarwinds give some help? this is not good and a major step backwards.... i love new features don't get m wrong, but my primary focus is 100% on performance increases to make solarwinds more scalable without having to buy a £250,000 storage array...

cheers

dan

rspoelstra

Just out of curiosity, do you monitor response times on these nodes? I am experiencing very similar symptoms after upgrading to 10.1. One thing I found is that when a device goes down for more than 1 polling cycle, it never shows that it comes up. On the NPM Engine Status I show 0 ICMP polls per second and we monitor the response time which has been flat-lined. To get a node back up, we either do a "Poll Now" from the manage node screen or just send an extended ping from the SolarWinds server until it comes back up. Just curious if you had heard anything from Solarwinds support on this issue.

FormerMember

i am now getting this issue too.... nodes come back UP again, i can get SNMP stats back from them, but PING does not come back, even though i can manually PING the devices from the command line on the polling server

when are SW going to comment on this LOL....

bshopp

We have a 10.1.1 Release Candidate which addresses some things and we are actively working with folks to investigate and address these type of performance issues. If you have not opened a support case, please do so we can gather data and investigate.

rspoelstra

I'm not sure if this will help you out at all or if its worth trying but it seemed to fix my issue that we were having here with nodes not coming back up. I have a ticket open and the dev team was seeing a lot of SNMP timeouts. We had them set to 2.5 seconds with 2 retries which has to be the default since we never changed it. Support suggested increasing this to a 15 second timeout with 4 retries. To do this make the following changes in the registry:

"HKLM\SOFTWARE\SolarWinds.net\SWNetPerfMon\Settings\SNMP Timeout" = 15000
"HKLM\SOFTWARE\SolarWinds.net\SWNetPerfMon\Settings\SNMP Retries" = 4

If this key not exists, create it (it could be either string or DWORD value).

This seems to have cleared up the issue and I'm cautiously optimistic its cleared up an APM issue we have been having for months as well. Feel free to give it a try.

sgreath

Very similar issues here, working with support on it. I've noticed my Orion Module Engine service stops then my volume utilization graphs and response time graphs stop working (along with web access). I start the service back up and web service is fine but the graphs don't update until I reboot the server. Happened after upgrade to 10.1 and have had it happen on 10.1.1 as well.