SNMP fails but device shows up.

lcsw2013

Has anyone seen this happens? It's almost as if the device gets stuck in time. I have a different devices from switches to servers, etc that will fail snmp but still display stale data and show the device as up (Green).

Many times a manual ping of the device actually shows the device as down which basically creates a false positive in solarwinds.

I've ran a query and discovered over 500 devices in my infrastructure not polling on SNMP and all show green. When it comes to servers switching many of the Windows servers to WMI fixed the issue and those servers started to monitor normally. But Unix or Linux devices or network devices that only accept SNMP, those remained broken. I had to manually either change the community string and or restart the snmp process on the devices to fix the problem.

But it still seems like a bug to me if snmp suddenly drops and the device still shows stale data, alerts on stale data, and shows as green (Up). I've tried the basic's. Clearing sdf files. Uninstalling reinstalling collector and job engines. I even went as far as basically redeploying my whole environment but the issue remains there. So not convinced it's a collector problem.

The issue for us has been founding using the script as a workaround. I run that script once every 2 weeks find the devices not responding to snmp and fix them. But I wanted to bring up this discussion for visibility and hopefully catch the eye of a developer or someone from solarwinds who can be notified about this.

Thanks!

Find more posts tagged with

SNMP

poll fail

Accepted answers

lcsw2013

This environment has way to many problems than it's worth to fix. I have gone a different route and I have setup a new deployment I have in test mode. So far testing against the new environment it's showing better results. Basically what we have done is we still kept it on ping poll but have built something similar to what was suggested on this thread. Using a combination of reports and Universal device poller among other data we manage to get the data we need to keep track of SNMP correctly.

I appreciate the help from the community!

All comments

pratikmehta003

Hi,

If you want to show the device also as down when SNMP is not responding, then u can change polling type from ICMP to SNMP from list resources option and make changes to your node down alert accordingly.

U mentioned about the query to check on SNMP not responding nodes- can u post or attach it here if possible?

dominguesh

The node status by default uses ICMP, basically, it will poll every 120 seconds,

if no reply STATUS = WARNING and Fast Polling occurs every 10s for 120 seconds.

If still no reply after those 12 polls STATUS = DOWN.

Under edit node, under SNMP credentials do you receive a failed test?

Which version are you polling, v2 or v3?

Did you receive any results from SNMPWalk against the device?

d09h

For situations such as the one you described, I found myself adding timestamps in charts to help identify stale data. In some cases is was my first/ best clue, since a Node Details page will look innocuous without clues like that.

lcsw2013

These are all good suggestions that I've looked into in the past but ended up not working for me. Unfortunately, Our infrastructure is setup in what I can only describe as a non-standard setup. This causes way to many issues and headaches with monitoring but it's been this way even before I was hired here. So most of the other administrators and groups are hesitant to make changes. It's been a fight to get things in order.

I say this because switching from icmp to snmp causes another problem. Due to some excessive security, randomly snmp packets are being dropped by firewall and intrusion systems. So every time a drop happens this would put my devices in either warning or down state.

There are also a great deal of out dated devices running outdated OS. Meaning occasionally the snmp daemon or process just gives up and stops working. This was visible on my Windows systems. And switching to WMI fixed that problem. But other devices are snmp.

Here is where I'm stuck. With ICMP ping the device shows up when snmp is down. Changing to snmp up or down monitoring means we could have more false alerts due to snmp troubles.

I am looking at redeploying the entire monitoring environment from scratch fairly soon. Just to remove any potential of this being a solarwinds problem. I'll update the thread if I can find a fix.

I do appreciate the suggestions offered. Thanks!

yaquaholic

We manage devices for all sorts of internal and external customers, who don't always update us when things change, like SNMP comm-strings and we too have been caught out by up devices failing SNMP polling.

So I set up a simple UDP SysUpTime (OID = 1.3.6.1.2.1.1.3) poller and then configured an SWQL trigger to identify any devices that fail this poll.

SELECT AssignmentName, CustomPollerName, DetailsUrl, DisplayName, NodeID, Status, StatusDescription, StatusIconHint, StatusLED, UnManaged, UnManageFrom, UnManageUntil, Uri

FROM Orion.NPM.CustomPollerAssignmentOnNode AS E0

WHERE E0.[Node].[SysObjectID] IS NOT NULL

AND E0.[CustomPoller].[OID] = '1.3.6.1.2.1.1.3'

AND E0.[CustomPollerStatusScalar].[Status] IS NULL

AND E0.StatusDescription NOT LIKE 'Unmanaged'

I use this then as a report, to view the current devices failing SNMP polling.

You could also set it up as an alert.

lcsw2013

I appreciate the help from the community!