Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Alerting on errors as a percentage of traffic/packets

I found this three year old post on the same subject but I wanted to find out if anything had changed since that time.

We have lots of international WAN circuits and having them receive errors is not an unusual circumstance. I really need to be able to monitor the errors as a percentage of packets sent so that we know when it's a problem and when it's not.

One of our business units runs a tool called Proactivenet and it does this monitoring automatically. I really would like to be able to provide this same level of visibility so that we can try to stay ahead of them and prove that our tool is as good as theirs.

So tell me Thwak Nation, is there a way now to do calculations in alerts that wasn't there three years ago?

Find more posts tagged with

Accepted answers

All comments

FormerMember

i have the same problem. I escalate number of errors issues to the provider and most of them will not even look at the errors until they reach a specific percentage of bandwidth.

Orion does not seem to be able to do this. the technology required seems simple so i am not sure why, but then i might be mis-understanding how it would be done... i imagined it woudl be something like an underlying equation that worked out number of errors, bandwidth and current utilisation and populated %errors

maybe anyway

MarieB

Hi All--

I'm going to route this to the PM for comment.

dperkerson

Any response yet?

ebradford

Let's see if we can figure this out together. I don't think the problem is with Orion's capabilities. The issue is with time lines and manner of data representation in the MIBs.

Interface Errors and Discards are available to you by default on an hourly and daily basis. The hourly and daily basis stats are calculated from polled counter information. Cisco counters merely count. Cisco does provide a few time-base numbers, such as 5 minute output rate. For example, it at 13:45:30, discards are 37511, and at 14:45:30 discards are 38802, then Solarwinds reports the difference of 1291 packets discarded in the hour.

However, interface traffic (packets per second, or bits per second) is an instantaneous (or at least very short time period) reading. So, to alert on number of discards per hour as a percentage of pps, just wouldn't be valuable due to incompatible time lines. Let's say that the number of discards per hour is 18000/hr, and the pps is 200 pps. You could multiply the PPS by 3600 to get a projected PPHr of 720,000. 18000/720000=2.5%. Let's say you want to alert when 12% of packets get discarded.

That's fine. But, due to the inconsistent nature of data traveling through an interface, a few minutes later, the pps is only 23pps. Multiply that by 3600, you get 82800 PPHr. Divide 18000 by that, you get 21.7% discard rate. The errors per time period didn't change, there was just a dip in traffic. This would cause an alarm of irrelevant nature.

Now, how frequently are you polling the node? Every 60 seconds? Every 3 minutes? That's a lot of data! Here's an idea of how we could accomplish this instead: write custom pollers that checks interface counters for (example) packets output and total output drops. Poll this by whatever interval you wish, I would suggest 1 minute. Then, you use the UPD Transform results. That's the basic idea.

However, implementation will be harder, and I'm missing a few pieces. It seems the solution here is to find a delta over time for errors or drops and compare it to delta over time for packets. The keys would be ensuring the time periods are compatible, how to poll, and how to calculate deltas. Most devices that conform to generic MIBs can use 1.3.6.1.2.1.2.2.1.20 for interface out errors, 1.3.6.1.2.1.2.2.1.19 for interface out discards, and 1.3.6.1.2.1.1.3 (sysUpTime) for a time delta. Since we are concerned with the passage of time, we don't need the actual date -- just a difference in time for the two measurements. however, if the errors and packets are polled at the same time frequency, then we could eliminate the need for a time delta. What I am missing is a MIB hook for interface packets out. I don't see it in the generic interface MIBS. And, since Solarwinds cleaned up MIBS, the Cisco device specific MIBS are no longer available. When we do a "sho int" command, we get some info like "22050108615 packets output, 11476944422430 bytes, 0 underruns". If we could poll the packets output, then we could calculate a delta on that and find a number of packets per time period.

The second thing I am missing is how to get deltas on custom pollers. The standard Orion interface pollers have a rate field which holds the last value. This would be very helpful, however the standard pollers aren't available in the custom poller transformation. I get an error when I attempt to use ${interface.rate}. I don't see that custom pollers have a rate feature. If anyone know how to get a custom poller to compare the current value against the previous value, we would be alomst done.

So, let's say for instance that we call the custom poller for packets transmitted ifPOut (when we figure out how to poll this). And lets say we fingure out how to calculate deltas. And we are polling the data on the same intervals. Then the solution will be a simple matter of transforming as so: {ifErrPcnt}={ifOutErrors.delta}/{ifPOut.delta}.

Once we have that, then the alert is simple "Alert when {ifErrPcnt} is greater than .20" or whatever your threshold is.

Now it's you turn. Above are a couple of things that I'm not sure how to do. Can you help figure out one or both of them?

ebradford

I e-mail Tesh Support and Bill was very helpful. I have now created several pollers which will give me the information we need after we figure out how to calculate a delta. In the items below, I abbreviated unicast, multicast and broadcast as Ucast, Mcast, Bcast, erspectively. These pollers are set up as Interface Pollers, Raw Data, Get, keep historical data, do not display on website. Here are the OIDs:

ifOutUcastPkts 1.3.6.1.2.1.2.2.1.17

ifOutNUcastPkts 1.3.6.1.2.1.2.2.1.18

ifOutMcastPkts 1.3.6.1.2.1.31.1.1.1.4

ifOutBcastPkts 1.3.6.1.2.1.31.1.1.1.5

ifHCOutUcastPkts 1.3.6.1.2.1.31.1.1.1.11

ifHCOutMcastPkts 1.3.6.1.2.1.31.1.1.1.12

ifHCOutBcastPkts 1.3.6.1.2.1.31.1.1.1.13

ifOutDiscards 1.3.6.1.2.1.2.1.19

ifOutErrors 1.3.6.1.2.1.2.1.20

I then created a transform for {ifOutUcastPkts}+{ifOutNUcastPkts} called ifOutPkts, and one for {ifHCOutUcastPkts}+{ifHCOutMcastPkts}+{ifHCOutBcastPkts} called ifHCOutpkts. I checked for equivalence by subtracting one set from the other set. On some interfaces, the result is zero, indicating equivalence. However, on an interface of one of our 6509 (which connects to a stack of 3750), the results are far from equivalent. So, I looked at the transforms separately, and compared them to a "show Interface" output. The one which reports sum of HC packets was a few thousand packets off – but was only .000000194% off – good enough for my use.

So, now we just need to figure out how to calculate deltas between two polling instances, and we are golden. So, if I poll every 5 minutes, how can I compare the ifHCOutPkts that just got polled with the ifHCOutPkts that got polled 5 minutes ago? This may be the hardest part though. Once we can do that though, the last transform will be

delta(ifOutDiscards)/delta(ifHCOutPkts) and compare that against the threshold. Almost done, and yet far far away...

I'm waiting to see what my new best friend Bill has to say...

ebradford

Well, bummer, no luck. I guess there isn't anyway to create a real-time delta using Custom Pollers or Transforms at this time. I have submitted a feature request.

If anyone can prove me wrong and show how to create a CustomPoller Deltathat can be used in an alert (graph use is unnecessary for this purpose), I'd gladly send a King-Sized Kudos or Whatchamacallit bar.