Inaccurate graphs over 70Mb

Question

Hi, I'm seeing a strange problem on solarwinds. This is happening on various nodes some servers, some routers but once the bandwidth utilization reaches about 70Mb the solarwinds graph then shows the bandwidth as 0Mb. I know the utilization on these interfaces is fairly constant going up or down by about 5 to 10 meg, so the graphs become completley inaccurate once the utilization goes up to high and I end up with a zig zag that looks like the network is completley shutting down every five minutes, which I know it's not.
Has anyone else seen this problem? and have an idea of a fix, I have the same problem with both orion 7.6 and 7.7.
Thanks
Chris

euphorik · Answer

Wow - Funny enough, I've had the exact same problem for the past 2 years which had me chasing global sync issues on some of my BB devices thinking this was actual traffic issues. My threshold was just over 60Mbps on all interfaces, which also led me to wonder about counter rollover, but I never noticed the 64bit counter checkbox on all my devices!

Brilliant. No idea why this isn't simply a default setting given most modern interfaces are operating at far above 60mbps. Surprised this never became a hot topic on the forums...

Thanks for the math BK.

shandcs · Answer

BK,  Thanks for that, switching on the 64bit counters has solved the problem on a Nokia firewall, a cisco switch, and a foundry iron switch. We also had a Netcache proxy showing the same problem for this one we had to adjust the sampling interval of the interface. So thanks again.
Chris

BK · Answer

My first guess at the problem would be the Counter32 rollover.

Your standard MIB II Interface Counters are 32 bit, so after 4294967296 bytes, the counter rolls back to zero.

That may sound big, but if you do the math, it means that a 70Mb/sec interface will roll the counter over in 8.1 minutes.  A 200Mb/sec interface will roll the counter over in 2.8 minutes.

So your interface polling rate would need to be faster than the rollover interval to prevent your data from looking like a random number generator.

Solution 1) Poll faster on those interfaces.

Orion does try to handle counter rollover.  If a counter has decreased in value from the last time it was polled, a rollver is assumed.  In the Advanced Setting you can set whether Orion repolls the interface or assumes a rollver and calculates the usage based on the 4GB rollover point.

Because interfaces were getting faster, the SNMP gurus of the world introduced the ifHighSpeed MIB which uses 64bit counters.  If a 64bit byte counter is used on an interface with 200Mb/sec of traffic, it would take 23,381 YEARS before a counter rollover.

Solution 2) Use 64bit Counters

In the Orion System Manager, you can bring up the Node Details and check the Allow 64bit counters box.  Orion will then tell all the Interfaces of that node to try and use 64bit counters.  If the 64bit ifHighSpeed counters don't reply, the interfaces will fall back to 32bit traffic counters.  You can see the status of using 64bit or 32bit counters on the Interface Details

Now the bad news.  It has been my experience that vendors do a poor job of supporting 64bit counters.  From counters that run backwards, to some that randomly jump 2G or 4G in between polls.  If you switch to 64 bit counters and your data looks worse, your devices has bad 64bit counters.

BK

BK
Nobody Special