Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

v7.2 False Positives

I recently upgraded from 7.1.15 to 7.2. After this upgrade I continued to get a large number of false positives from Solarwinds. I changed the ICMP timeout to 5000 and byte size to 0, hoping this would solve my problem. It did not. I then changed the Polling rate to 5 minutes. This still resulted in a large number of false node down events.

Is anyone else experiencing these issues?

I have since rolled back to 7.1 and everything seems fine as before. I guess I'll wait a little longer next time before upgrading.

Joshua E. Morehouse
Senior Network Engineer
Friedman, Billings, Ramsey Group, Inc.
"Capital For Your Conquest"
jmorehouse@fbr.com

Find more posts tagged with

Accepted answers

All comments

rharland

I'm experiencing issues very similar to what you describe, but only with nodes that are on the same subnet as our Solarwinds box itself. Very strange! Perhaps we should open a case?

thanks
Russ

Network_Guru

What is your fast polling timer set to?
I have mine set to 240 seconds, which in addition to the 300 second icmp polling interval
seems to give very few if any false positives.

You might also want to try continuous pings to some of these devices to verify there is actually no packet loss.

-=Cheers=-
NG

FormerMember

NG - I am experiencing similar issues with false positives on Enterasys switches. Where is the fast poll timer setting?

keist

Are you using Cisco 6500 series routers/switches in your network?

Dan

Network_Guru

In the System Manager app go to:
File > Advanced Settings > Node Warning Interval

I have mine set to 240 seconds which should allow at least 8 more polls of 2 pings each
(fast polling) before the node gets marked as down. Note, this will increase the time
for receiving legitimate outages, so you should balance this according to your alerting needs.

-=Cheers=-
NG

cshiver

If its applicable, have you verified that there are no hung terminal-services connections to this server?

- Chris

FormerMember

Thanks NG! I missed that.

pancamo

I also have the same false positive issue with ONLY nodes on the same subnet and the solarwinds box itself. This must be a bug of some kind...

quote:Originally posted by rharland

I'm experiencing issues very similar to what you describe, but only with nodes that are on the same subnet as our Solarwinds box itself. Very strange! Perhaps we should open a case?

thanks
Russ

SSonnier

I am having the same issue with some devices on the same subnet as the Solarwinds Orion Server. Cisco 3500' series and 2900 series switches only are reporting between 50 to 80 percent packet loss....

Network_Guru

They have fixed the false positives in Version 7.2.2 which is a production release.
I had the same issue with 7.2, although I still see groups of nodes show up as
"not responding" (they seem to have done away with the "node status is warning"
description in 7.2.2

Let me know if this fixes your issues, as I am still working with Solarwinds on this issue.
I already noticed the problem with nodes on the same subnet back in V7.0.5.
The polling engine may be flooding the local subnet with ICMP packets when the first one
fails, due to the server's arp cache timing out, and then the switch may be limiting
this flood of packets? I even cut down the polling on the locally connected nodes
to 3 minutes, which is less than the default 5 minute arp cache timeout, but this made
no difference.

-=Cheers=-
NG

jonchill

Since upgrading to v7.2.2 I have noticed more false positives for 3Com switches my fast polling time is set to 300 secs and most of these are on the same subnet as the Orion servers the only ones that are not are ones that I know are dropping packets.

This problems seems to be on going since v6.5. Do Solarwinds see it as a bug in Orion and if so shouldn't they be doing something about it?

Thanks

Jon

IT Infrastructure Manager
Pilgrim Hospital
UK

bleearg13

I am experiencing a somewhat similar issue, except the alerts are not due to nodes that Orion thinks are not responding. For some reason, Orion feels compelled to notify me sometimes of nodes that have rebooted, except those nodes rebooted days, weeks, or months ago. Do you think this issue is related to the false positives that some users are experiencing?

-evt

Network_Guru

Bleearg,

I don't see that as being related. You must be running an older version of Orion.
Since V7.? I now get the reboot alerts within minutes of the device rebooting.
Previously you would only get the reboot alerts when the device rediscovery took place,
according to what you have the interval set to.

-=Cheers=-
NG

bleearg13

I am now running 7.2.2 and just received an alert regarding a router that rebooted on December 7, 2003. There doesn't seem to be any correlation between the times when I receive the alerts and anything like high CPU utilization on the Orion server.

I'll probably just live with it due to the very frustrating responses I've received by emailing support@solarwinds.net. Do they even read the questions before sending out their formula response?

-eric

Haley

Eric,

Reboot alerts that refer to an old date are typically related to either a faulty counter on the device or a redundancy setup. To verify if the counter is having issues, simply open the SNMP Graph tool (available in the Engineer's Edition Licensed/Eval), select to monitor RFC1213:SysUptime with No Calculations selected, click on the "Custom" button, check the box for "Log data to a file as it is collected", and click OK. Then the next time the reboot alert is reported, simply look at the log file of the SNMP Graph. If you need assistance interpreting the results, simply email them to Support@SolarWinds.Net.

If another device is set up to come online using the same IP as a device that just went down, then the Orion software is going to see the two devices as the same device. Each device rebooted at different times, and this may be the reason the reboot alert is triggering.

jonchill

I've logged a call with Solarwinds support as I'm getting fed up with the false positives and my boss is starting to put pressure on to resolve the problem.

I'll let you know if I get anything worthwhile out of Solarwinds.

Jon

IT Infrastructure Manager
Pilgrim Hospital
UK

Network_Guru

Jon,

One way to fix this, is to capture ICMP packets with a sniffer to see what is
actually happening on the ethernet segment your Orion server is on. You may have a
problem on the segment, or it may be a legitimate problem with the polling engine.
If the latter, you could e-mail the sniffer capture in .csv format to Solarwinds
to prove your case. It's best to filter on just one device which is experiencing problems.
Try to include only the relevant data (ICMP) with about 20-30 minutes of packet capture
around the time when you see the false alerts.

I've been out of the office for the past week, but will be examining some more captures
this week, to try and interpret how the polling engine behaves when packets are dropped.

-=Cheers=-
NG

bleearg13

Haley,
Thanks for the suggestions. I do wonder why this has just started happening, is that problem (false alarms on old reboots) something that 7.2 has begun to monitor or account for? I have had Orion running for several months now without these errors and it has suddenly just cropped up recently, since the upgrade to 7.2.

-eric

pancamo

Upgraded from V7.2.0 to V7.2.2 and V7.2.2 appears to have resolved the false positive problems with polling server/switches on the same subnet as the polling engine.

Thanks!

Dan

toko

I'm seeing similar issues. Most noticeably with Cisco Catalyst 3500 and 2950 switches. Response times look good and most of the time I have a little blip on the %packet loss graph at 10%.

Most prevalent in early morning hours when traffic is very low.

All these switches have fiber trunks that lead back to a Cisco Catalyst 4506. All of these switches are on different subnets from the monitoring server.

So things that are in common, Solarwinds box, Central Catalyst 4506, and Cisco products.

I have seen other products with "In warning" alerts, but everything leads through a Cisco switch.

I'm a little confuse why I'm getting alerted. I can understand an event, but if it really is the %packet loss at 10 percent for a brief period why set off the alarm, especially if it is based on an icmp poll?

I'm also seeing alarms on everything at bootup or restarting the service. I don't remember having this issue with version 7, but I certainly had it on 7.2 and continue to have it on 7.2.2

I tried tweaking solarwinds settings. It is so sporadic. Makes me wonder if it isn't something to do with the network or the 4506.

Any progress on this?

Any way to roll back to an older release?

T.K.

toko

One other thing I've noticed on the "in warning" messages. I get an e-mail alert, but it doesn't appear in the active alerts.

If Alerts are set to send e-mail and an e-mail is generated, shouldn't it be listed as an active alert?

T. K.

iunderwo

Active alerts only stay active when the alert condition is present. After it resets, it disappears.

I don't even consider warning conditions to be a valid trigger or reset for that matter since they seem to be transient. It may be a prelude to an event, but not alert-worthy. I still see tranient warnings from time to time, but they always disappear.

I'd remove them as an alert condition.

// Ian Underwood - Service Management
// Level 3 Communications

SSonnier

I upgraded from 7.1.15 to 7.2.2 and the issue went away for ~ 1 week. I am now having the issue again on the same subnet as the Orion Server. Cisco Switches are reporting 30 to 50% loss. What now? I staggered polling between International and Domestic sites, tried changing the ICMP Timeout to 3000ms...nothing is working.

SSonnier

Alright.....
Changing the polling increment on the Switches on the local LAN seems to have fixed my issue. I shortened the increment from 120 seconds to 20 for ~15 devices. I hope this does not have an ill effect. Packet loss has subsided from 50-70% down to 0% for devices on the same subnet as the Orion Server....

Network_Guru

One possible explanation for this behaviour could be the arp cache size on the Orion server.
Since the nodes you are polling are all on the same segment, the server must cache the
MAC address of these servers if it is to get an ICMP reply packet from the first ping
sent out. If the arp cache on the server has cleared, then the first ping is usually
dropped while the MAC address is discovered from a broadcast arp on the local segment.

I was polling every 5 minutes, which is also the default arp cache timeout value for
MS Windows. I set the polling frequency to 2 minutes, thinking this would resolve the
issue, but it did not. I could see the node in the arp cache on the server using "arp -a",
but it still dropped polls and indicated packet loss in the 50-70% range.

Magically, the packet loss stopped sometime in March, so I never did figure out what
the cause of this was.

If you have a sniffer available, I suggest you put it on the server's local segment,
and find out what is actually happening "on the wire".

-=Cheers=-
NG

SSonnier

Well I thought lowering the polling increment was the answer...but it was not...still showing the same high % loss arghhhhh.....

pancamo

We have the same problem again on the same subnet... A few days after upgrading from 7.2.0 to 7.2.2, the problem is now back again... the devices on the same subnet are having 30-60% packetloss again!

quote:Originally posted by SSonnier

I upgraded from 7.1.15 to 7.2.2 and the issue went away for ~ 1 week. I am now having the issue again on the same subnet as the Orion Server. Cisco Switches are reporting 30 to 50% loss. What now? I staggered polling between International and Domestic sites, tried changing the ICMP Timeout to 3000ms...nothing is working.

pancamo

I have a Opnet sniff from the box. ICMP packets are not being sent (30-50% of the time) to the local subnet devices, no packets! When the poll are working correctly, I see ICMP packets every 2 minutes...

quote:Originally posted by Network_Guru

One possible explanation for this behaviour could be the arp cache size on the Orion server.
Since the nodes you are polling are all on the same segment, the server must cache the
MAC address of these servers if it is to get an ICMP reply packet from the first ping
sent out. If the arp cache on the server has cleared, then the first ping is usually
dropped while the MAC address is discovered from a broadcast arp on the local segment.

I was polling every 5 minutes, which is also the default arp cache timeout value for
MS Windows. I set the polling frequency to 2 minutes, thinking this would resolve the
issue, but it did not. I could see the node in the arp cache on the server using "arp -a",
but it still dropped polls and indicated packet loss in the 50-70% range.

Magically, the packet loss stopped sometime in March, so I never did figure out what
the cause of this was.

If you have a sniffer available, I suggest you put it on the server's local segment,
and find out what is actually happening "on the wire".

-=Cheers=-
NG

Haley

With recent development in Virus Scanning software and Personal Firewalls due to worm attacks, a shim is being inserted into the IP stack by the monitoring software. Simply uninstalling the software does not remove this shim. I recommend uninstalling any Virus Scanning software or Personal Firewalls from the Orion Server. This may resolve the packet loss issue.

SSonnier

Hello,
Only Solarwinds Orion is installed on the Dell Poweredge 2650 Server. The only other software installed is Dell OpenManage.