Interface Percent Utilization charts: Real Data vs last hour vs Details " Current Traffic " ?

I have read that the standard " Percent Utilization " chart is an average and the "Real Time Data" is I suspect close to real time. I still have a question.

Under the Interface details " Current traffic " shows 1.321 Gbps receive. The Percent Utilization chart show a nice flat line under 20%. The Real Time graph shows plenty of spikes above 80%, some above 100% and an occasional spike over 120%.

This interface is one port in a two port "agregate interface" and the other port shows very similar data. That means on a 20gig port pair ( 10gig each ) I have losts of spikes over 60%, and good amount over 80% but on average both are under 20%

That seems to be a lot of data that averages out to not a lot of data. Am I reading this correctly? There are no custom speed overides in the port details.

How can I get any percent utilization over 100%?

Find more posts tagged with

ntm

npm

Accepted answers

All comments

mesverrum

In the real time polling you can run into quirks of polling cycles and the refresh rate of the counters themselves. In ye olden days lots of lower end Cisco gear wouldnt even update the SNMP counters more than every 15 seconds or so, so if you hammer it every 1 second you would get exactly the same byte sent count 14x and the suddenly it would spike and then it would stay stuck again for the next 14 attempts. Solarwinds, along with most other SNMP based tools, started doing the thing where they subtract the previous polled value from the current value, and then divide by the seconds between the two timestamps for their charts to only show "new" byte traffic. If you are in a situation where the real time poller is hitting your device slightly offset from the actual refresh rate of the hardware then you can get cases where most polls show low utilization, then you get a sudden "burst" which is really just the hardware's internal snmp counters finally moving again. If solarwinds trusts those timestamps (to be honest sometimes it shouldn't, especially if the hardware has high-ish latency or slow snmp response times) then Orion assumes every poll should show new data then you can see utilization rates that seem to sawtooth and burst above 100%, but if you use a slightly slower polling rate it would probably show something that more accurately represented the running average data transmission speeds really happening on the hardware. It's SUPER important to remember that SNMP is a bit of an abstraction and rarely is the honest to god real time truth about anything happening on network gear.

Hardware chips in the interfaces "do the thing" as fast as possible, in the process of doing it they try to take notes about what they did and whenever they get around to it they send a stack of those notes over to the snmp agent who then waits around for someone to ask it what the interfaces have been up to, it looks at its notes and tells you "last time they dropped off the notes they said they did this" and if you ask them how long it's been since those notes came in they have no idea and just say "well right now I think its 6:19, so i guess it was earlier than that" and then whatever software is doing the polling has to figure out for itself how much it trusts that SNMP agent, because in honesty it doesn't seem like it even knows what bytes are, its just a secretary relaying whatever nonsense it saw in the memo.

In the advanced settings you can actually set a value for how fast the "real time" polling is, so I would suggest you slow roll this from 1 second to something higher, maybe 5 or even 10 seconds depending on your hardware and see if the charts become more believable.

[mention:081a4c1172404696b328dc0ea4b091e5:fb5d84b10a5745448a7a45dafc1faa43]

John-AVUSD

I guess in 5 years the location of those settings has changed. The path mentioned in the thread does not exist on my server running 2023.4.0.

In the GUI, I have the following path:

Admin
Thresholds & Polling
SolarWinds Platform Polling Settings

I don't see " Live Polling " any where on that page. The polling interval is 120 sec.

This brings up another question. If I create alerts based on unexpected change in traffic and the "Real Time" info is not accurate, will I be getting false alarms? ex: if I want an alert to tell us that the transmit date has suddenly jumped because our data is being stolen by some hacker, what data stream should I trigger that alert on?

mesverrum

There is no GUI method to get to the advanced settings, its an "if you know, you know" kind of hidden menu. If you try to copy his link you might be getting jammed up on the fact that he used http instead of https. I usually log into my platform and delete everything after my servername from the url bar and just paste in the path at the end. I can confirm it totally exists and works in current versions, also if its not obvious it would only be accessible to full platform admins.

/Orion/Admin/AdvancedConfiguration/Global.aspx

Regarding alerts, you can't alert on the real time data (maybe you can with custom SQL if you are really motivated), it doesnt get stored into the historical tables where alerts check. The alertable tables are based on the interface statistics polling interval, which would be every 9 minutes by default. At that slow of a polling interval this is a non issue, and you should never be setting alerts that trigger on a single polling cycle anyway because thats going to be spammy and irrelevant in almost all cases.

The computational load of doing real time polling and alerting on that data in second by second increments is pretty extreme and most Orion customers don't have the appetite to invest the money in hardware or cloud resources it takes to make that stuff happen at that level of granularity. Everything sounds like a good idea to set things up that way until the finance department realizes what it costs to run that sort of thing for a non-revenue generating activity.

If you are worried about your data being stolen by some hacker then trying to infer that action from a few seconds increase of network activity is among the least conclusive ways to know anything about what happened. How do you reliably turn a raw count of bytes into something that can differentiate file exfiltration versus the traffic of an admin doing a remote pull of log files or a system backup or one of any other hundreds of legitimate network activities.

John-AVUSD

mesverrum,

I don't intend to take action on a few seconds of high activity. My goal is to get an alert to look at the Solarwinds data and then determine if we should look at the server info it's self. My idea of getting the trigger on the "Live data" was to try to get the quickest " hey you might want to look at this... " alert. If I could even get a poll of a minute of abnormal traffic, that would be better then none.

If the Live view is not accurate enough for monitoring, I'll let me boss know there are too many variables in the mix to base any action on and not use it in a dashboard the department uses.