It looks like we have solved the mystery as to why we were seeing such a difference in the numbers being reported by MRTG when compared to SolarWinds. To recap, the MRTG numbers seemed to cap (always) at just under ~120Mb/sec. After some more searching we discovered that the difference is due to the 32-bit counters in the 5 minute interval (and if what follows is not correct, I would be interested in any feedback). The maximum amount of traffic the 32-bit counter will capture for the 5 minute interval is just shy of ~115Mb/sec. Here is the formula we found to support this claim:
(2^32) / (5*60) = x (where "x" is the maximum amount of traffic)
4,294,967,296 / 300 = x
x = 14316557.65333
Now, the formula we found said that "x" was the bytes/per second and that you then multiply this by 8 to get the BITS/per second. That was a bit confusing because we thought that 2^32 was bits, but we went with the formula as written. When doing this we end up with ~114.5Mb/sec. Ironically, this is where every GIG port we are monitoring maxes out when we look at the 5 minute interval of MRTG data (well, the graph actually shows 120Mb/sec, but you get the idea). There were a number of posts by individuals who were having similar issues with MRTG and GIG ports when using the 32-bit counters.
One of the challenges we ran into was how to determine the counter: was it 32-bit or 64-bit. According to the MRTG web site it came down to the SNMP setting in the Target string inside the mrtg config file. If the string is the following:
You are running SNMPv1 and by default you are using 32-bit counters. If, however, the string was set to the following:
You are running SNMPv2 and this will poll the MIBs for use with the 64-bit counters. There is more on this at the following link actually: http://mrtg.meulie.net/doc/mrtg-reference.en.html
I would be interested in any feedback on this theory/formula used for the conclusion that 32-bit counters in a 5 minute view max out at right around 115Mb/sec, but it is clearly what we are seeing on our end and thank God we have SolarWinds or we would never had stumbled upon this fact. Okay, hope this helps someone out there...