Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Path Echo Problems?

Hello,

We recently setup a few IP SLA's for Path Echo. We're seeing some strange results. The operation shows up but when I look at the results, its showing 1 hop. (To the MPLS Edge) I've let it run for a while and every now and then it'll show the proper 3 hops. Trace routes work fine from the router. I have this happening at 2 Sites back to our NOC but from our NOC to the two sites the operations are fine.

Has anyone else run into this?

Thanks!
Eric

Find more posts tagged with

Accepted answers

All comments

michalB

Hi,

you might check the operation results directly on the router and compare the results with the ones displayed by IP SLA Manager. Or you can open a support ticket and reference this thread.

ecornwell

The result on the router is the same, just one hop.

michalB

IP SLA Manager reads just the data that is read from the router. Therefore the issue is not in the IP SLA Manager, but on the router. Did you create the operation manually on the router, or did you create it using IP SLA Manager?

JesperVestergaard

I have the same issue on a few of our routers.

Any clue as to how you solved your problem ?

Mý probe is configured manually (don´t have admin rights to create the probe - I do have the needed enable level access, yet restricted by TACACS only to perform some commands)

ecornwell

I only worked on it for a little while after posting. I just checked and it is still broken. From what I can see the router isn't doing something correctly. I've been too busy to open a TAC case.

JesperVestergaard

Darn... !! - I hoped you did my work for me

Anyway, i think it´s related to multiple SVI´s on the router - Or even maybe the router platform. My "problem router" is both 2800 series - I never saw this problem on switches or 3800 series routers.

Anyway - I´ll keep looking, and post if i find the answer.

ecornwell

Sorry!

FYI, all 3 routers we're using are 2821's.

michalB

so we actually got into the similar situation in our lab: the traceroute command on the device displayed correct results with about 20 hops to google.com, but path operations displayed just a few hops (2 - 3).

the guy taking care of the lab found out it was a firewall issue. This is what I got as explanation:

Icmp port unreachable was blocked at the exterior router.

When you are using a unix/cisco system for trace it will push out udp packets with increasing time-to-live numbers. These are sent out in triplicate with the first port number usually in the 32000 range. When it finally hits the system there is an assumption that there won’t be a service listening so you get a port unreachable reply rather than a timeout.

Because port unreachable is something that a hacker might use to scan your network for open ports, some firewalls and systems are setup to not reply to them and can cause issue with the trace.

Does this help?

JesperVestergaard

I´ll try to investigate this. - Tnx.

but... In my setup, a C3750 switch is directly connected to the routers LAN. When configuring the PathEcho on the switch, the probe works fine, and solarwinds obviously also to collect the statistics fine.

This (in my oppinon) does not support the idea, of ICMP unreachables beeing disabled on the PE router ?

michalB

Well, you seem to by right. But if the device itself is not able to perform the traceroute, there has to be something wrong with your network.

JesperVestergaard

Hi all

It appears that the implementation of IP SLA is “slightly” different in routers and switches.

Not related to the router platform (that I assumed it was) – but to the fact it´s a router…!

What we have discovered so far, after sniffing our traffic:

When the switch executes the EchoPath, it does so as described in the documentation

· Sending a packet to the selected destination, with a ttl of 1

· getting a response from next hop with ttl time exceeded

· sending a new packet to the selected destination with increased ttl of 2

· etc,….. until it reaches the destination.

· Execute ping, towards each hop of the discovered trace

When the router is executing the IP SLA, it does so slightly different:

· Sending a packet to the selected destination, with a ttl of 1

· getting a response from next hop with ttl time exceeded

· sending a new packet to the discovered next-hop with increased ttl of 2

· etc,….. until it reaches the destination.

· Execute ping, towards each hop of the discovered trace

Now, why is this a problem in a MPLS network ?

Well, most providers protect their management interfaces from customer traffic, like UDP etc, hence the IP SLA implementation of “UDP echo probing” for the path, does not work. Hence.. The result is a ICMP unreachable response.

Normal trace traffic is normally not blocked.

I´m not saying that this is the only and conclusive reason – I (we) might be learning something new, when the TAC case created, hopefully will reveal something new – Or confirm my findings.

One extra thing – In the packet from the router, there is also an option included in the header:

Loose source route -> “the selected destination IP”

I don’t know if this might (also) be related, or of this is used to carry the information to the discovered next-hop, about where my traffic wants to go..

Anyone having input ?

Anyone able to recreate the behavior – or create a EchoPath probe that traverses a MPLS, that actually works ?

bob55

Hi Jesper,

Did you set right VRF name for your operation?

Robert

JesperVestergaard

Hi Robert

My IP SLA router is not vrf avare. It might be a router deep in a topology "somewhere" and the traffic simply traverses a MPLS network, to reach the destination.

It might even traverse 2 MPLS´s (Global MPLS for the uplink, Local MPLS within a country)

JesperVestergaard

Finally we managed to debug the error, and reproduce enough documentation to satisfy TAC. The error is caused by LSR (loose source routing) beeing enabled by default, once path probes are defined.

The reason for different destination addresses in UDP packet from the router is that the LSR (loose source routing) ip option is enabled. On the switch this option is not in use, therefore the destination address for all UDP packets is the same.

If the IP SLA path probe is traversing a router which is not recognizing the loose source routing option, it´s sending the packet to its L4 udp handler. At this point, the intermediate device returning a 'U' (destination is unreachable) and the probe stopped at this point.

Cisco confirmed our findings, and created the bug id: CSCtj56321

Once this problem is solved, and the "minor" issue of solarwinds requiring enable login to devices for path probes is fixed, i´ll implement IP SLA probes on ALL my MPLS CE routers.

JesperVestergaard

FYI, Cisco released an expected fix date for this bug; It is scheduled for the 31/03/2011.