Polling - Detecting Degraded Circuits / Packet Size

Question

If a solution to this has already been posted, please point me in the right direction, as I've been unable to find what I'm looking for via searching.

We are running into a problem detecting degraded data circuits - circuits that drop a high percentage of "full size" (1500 byte) packets.  It fortunately doesn't happen that often, but when it does it makes us look bad.  The main problem we have is that regular polling / pinging will make it through without issue, only dropping a few packets here and there, but once you try and push a file transfer through or ping with 1500 byte packets, the circuit will drop 20-50% of the packets, making it unusable.

We are currently running NPM version 9.1 SP5.  What have you found to be the best way to detect this problem with NPM?

Thanks

nite · Answer

1500 byte packets make it through without issue on a "regular" day, so there doesn't seem to be an MTU issue that we can see - we just use it as a test when our users start complaining about slowness, network applications not working, etc.

For the most part I think we're lucky with our carrier for problem resolution - when we call in with these sorts of issues, they are usually able to identify the problem resonably quickly (once they pickup the ticket of course 
 ) and usually it's just a reset of their equipment or an adjustment of settings and things are back to normal.  All in all this inquiry came from a manager's request for us to be more "proactive" than "reactive" in these cases - in pretty much all other cases NPM has us covered, it's just this scenario we have problems with.  Fortunately it doesn't happen that frequently.

Thanks for the additional troubleshooting steps bleearg13 - this could definately come in handy in the future if we're unable to get a resolution to a DSL related problem.

bleearg13 · Answer

That's definitely a sticky situation to be in.  If the problem always shows up when pinging with 1500-byte sized packets, and the loss is not 100%, at least you're likely not dealing with an MTU issue.  If you are seeing 100% loss on 1500-bytes, then you might be dealing with an MTU issue within your carrier's network.

If the carrier won't let you manage the DSL routers at your locations, then it's their responsibility to find the cause of the packet loss.  I would provide to them as much information as you can:

* Ping your LAN gateway with 1500-byte sized packets.
* Ping your WAN gateway with 1500-byte sized packets.
* Perform a traceroute to some website and attempt to ping one of the hops in the path with 1500-byte sized packets.
* You could run MTR on a linux box, or pathping from Windows and gather similar information.
* As mentioned before, you could set up a pair of routers or switches on each end of your circuits that do IP SLA between them.  Orion can be configured to monitor the IP SLA MIB for threshold failures, or depending on the capabilities of the equipment, something like Cisco's Embedded Event Manager can notify you via SNMP or syslog.  Juniper has a similar application called Realtime Performance Management, if you use their equipment.

As a carrier, the above information is what I would like to see from a customer when they report a problem.  While ICMP is not a foolproof way to identify a problem, it's a darn good start.

Finally, I'd like to mention that DSL circuits are a pain in the a** to deal with.  DSL circuits are by far the most unstable circuits we provide.  They are extremely prone to errors, since they are limited by distance and physical copper quality.  Good luck and hopefully some of this rambling was useful to you.

nite · Answer

Yann, I doubled checked our Errors/Discards graph for the circuit we had problems recently and it doesn't look like Orion seen anything - I'll explain further.

bleearg13, the circuits we're having troubles with are managed DSL circuits.  The carrier's equipment onsite is the router and all we have onsite for managed equipment is a Cisco switch, specifically monitoring the port between the carrier equipment and our switch.  The problem we have with catching errors, etc is that the problems are happening on the carrier's equipment and not on ours - the interface between our equipment and the carrier's is clean.  The circuits are not being maxed out, etc during these problems, as we do have those problems from time to time and even regular polling starts to hiccup so it's easier to anticipate what might be happening in those cases.

I was hoping maybe there was a way to adjust polling to "catch" this problem, as pinging with 1500 byte packets reveals the problem, but otherwise it's hard to tell the problem is happening.  This may not be as realistic as I first hoped, but I'm always open to suggestions.

The comment about IP SLA with the carrier does bring up a good point though - if Orion doesn't have a realistic way to monitor this, we may be able to get the carrier to closer monitor this problem by requesting it or worst case paying a little more (just thinking out loud here).

Thanks for everyone's feedback so far.