Demystifying Packet Loss Monitoring in NPM

Question

Hi,

I've been using Solarwinds for some time now and just can't get my head around how to set up packet loss monitoring so that I get meaningful alerts (i.e. there is a problem that I need to look at). I've done a fair bit of digging and after reading a few articles here and in the knowledge base, i think i have an understanding of how Solarwinds works out packet loss but i can't figure out how to alert based on X% of packet loss without getting alerts when there isn't an issue. I guess i'm looking for both confirmation that what I've put below is correct and also what I could do to get some meaningful alerts. Apologies if this information is out there but I can't find it.

So how I believe it works:

NPM polls the node based on the polling interval you configure (120 seconds by default), i set mine to 15 seconds. This sends 1 ICMP packet/ping???

If this ping doesn't receive a response, or a response from however many retries you have configured it for, then it goes into 'fast polling' mode which will send one ping every 10 seconds for 120 seconds??? - Fast Polling

NPM calculates its packet loss based on the average of the last 10 pings so if 1 of these pings drops, that is immediately 10% loss??? - How is % Packet Loss calculated?

Now, my alert is configured to tell me if packet loss goes above 5% and remains above that for 3 minutes which i think is more than reasonable. Anything above 3% packet loss is too much for me when pinging from a Windows OS. I've changed the ICMP data that NPM sends so it should be the same packet size that any Windows OS should send by default, just so we are playing by the same rules - https://support.solarwinds.com/Success_Center/Orion_Platform/Knowledgebase_Articles/Packet_Loss_on_Orion_ping_command_does_not_have_packet_loss

If everything above is correct, how I am supposed to get alerted when there is actual sustained packet loss of 5%, not just when a single ping during fast polling is dropped? Can you change how many pings it sends in fast poll mode and how many pings it calculates the packet loss from? From the sounds of this article, you cannot - Orion Fast Poll and Node Statuses Explained - SolarWinds Worldwide, LLC. Help and Support

Many thanks for any help/advice you can give

mesverrum · Answer

So I think you have a disconnect in your head between what Solarwinds is counting as packet loss and the metric you have in your mind.  Solarwinds is not counting overall packet loss on your server, at all.  It is saying 10% of the time when I tried to hit this device with an ICMP ping (yes a single packet) I did not get a response.

Each attempt is really a binary value, response / no response = 100% / 0% but the GUI shows you the rolling average of the previous 10 attempts.

So when a node goes down the display would step through from 0-10-20-30-40-50-60-70-80-90-100% packet loss over the course of about 100 seconds.  In the database this will actually shop up as just a running series of 100% packet losses.  When it comes back online it will do the reverse and the database will record 0% packet losses for every attempt after it comes back online.

That behavior is all pretty much locked in, and you have almost no control over it.

What you are asking is if Orion will tell you if a Windows server is dropping a long running average of 3% of all packets. Orion isn't tracking that normally, Orion is recording it's own success rate at pinging the target.

Now the rules of averages would say that if your server was dropping 3% of all packets then over a long run Orion should also have about 3% of packets lost, but if you look at the loss charts in the GUI that would show up as a device that semi-regularly alternates between 10% and 0% packet loss over the course of many days.  You could probably put together a report that says show me all devices with average packet loss >0 and <100 but never went down over the course of a month and use that as an indicator of packet loss averages but that's a really indirect measure and would be skewed by any downtimes or server reboots and just because Orion can't talk to a server doesn't always mean the server is dropping packets, maybe you just had a bad firewall rule implemented for a bit last week or even just a cut over to a new circuit.  There are a lot of confounding factors there that make this unreliable at best. If you have NPM only then thats probably about as good as you will get though.

Assuming you have SAM, I believe the best way to track real ongoing packet loss for a server is just to set up a component monitor on the Windows perfmon counters for tcp/ip segments sent/sec and restransmitted/sec.   This is the most direct and reliable way to keep tabs on the metric you are asking for, from that server's own point of view.  On the alerting side you'd probably need to do some math to calculate a percent for your threshold but thats where I would be doing it, completely ignoring the percentloss value.