cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 9

VNQM 4.1 Jitter and Latency Appear to have swapped

Hi All I will be logging a support case about this tomorrow, but thought I would ask the community, we have VNQM 4.1 and are running a number of IPSLA's. It appears that the jitter and Latency values of the IPSLA's have swapped, in that we are seeing latency of 4 milliseconds at sites completely on the other side of the country (Australia) and are seeing supposedly consistent jitter of 40ms everywhere, which makes no sense.

Has anyone had this issue before?

thanks

0 Kudos
14 Replies
Level 7

Any update on this issue? I updated my account settings to default and that didn't seem to correct the issue.

0 Kudos

For us this turned out to be due to a Cisco Bug on 15.x IOS routers, it was not a Solarwinds issue. That might be the cause for you as well.

After 42 days the graphs would swap due to the cisco bug

I was unable to find it through Cisco Bug Search either. I'd be interested in the Bug ID as well!

0 Kudos

Can you tell us the bug ID so we know what to watch out for?

Thanks.

0 Kudos

CSCuh17594

The bug contents:

Description

Symptom:
The round trip time value reported by UDP jitter probe shows spikes of 100+ millisecond when destination interface is tunnel or other software based interfaces and high CPU activity occurs on responder.

Alternatively, the bug may manifest itself as showing very low RTT values, such as 1 ms, for links that are known to have a far longer RTT

Conditions:
This bug has a variety of conditions. It may been when:
Uptime of the device reaches > 49 days
and/or
The CPU of the device is busy with various tasks when processing the UDP Jitter packet and determining its timestamp and the destination interface is software based
and/or
When the fractional timestamp of the packet from CEF is not available, for various other reasons.

Workaround:
If the only trigger for this issue in a customer environment is the 49 day uptime, a reload may resolve the issue temporarily.

However, in many customer environments there is other trigger conditions that this will not resolve and the only solution is to upgrade to a fixed release. Hence this is the recommended course of action in all cases.

Further Problem Description:
This bug is partially due to a regression caused by bug CSCuc74187. Hence you are more likely to see this issue in IOS releases with the fix for that bug.

However, it also includes other improvements to IP SLA timestamping which will resolve other causes of bad RTT values

It also should be noted that to obtain accurate IP SLA timestamps, it is advised to do the following:
Ensure both source and responder are synced via NTP, with a low offset value

The NTP server preferably should be equidistant as possible to both devices

On platforms that support the command, such as, the ME3400/3600 switches, ensure the "ip sla enable timestamp" command is configured

On ASR1000 source devices, ensure that you are using a IOS XE release which supports QFP Timestamping, such as 3.7S and later and configure it using the "optimise timestamp" and "precision microseconds" options under the probe

=========
There is a long list of fixed releases. Its not just 15.x,, but also 3.x, 7.x,(NX-OS?) etc.
And for those whose heads still ache, and whose walls have dents, Cisco notes that there were 130 cases related to this.
One has to wonder if Cisco themselves monitors their own networks.

cejay79 -

You are brilliant! Or maybe your use of a browser is brilliant. In either case, we thank you.

=Foon-

0 Kudos

No worries, but we only found this out after logging a tac case on it, as it had us stumped.

0 Kudos

For things like this, I usually try a couple of additional steps:

1. Restart all Orion services via the OrionServiceManager, and retest

2. If step 1 doesn't work, re-run the Config Wizard

The idea is that some settings are only checked when services start, and other settings may depend on the Wizard.

Finally, if all else fails, open a Support Case. There's a million ways for something to fail, and only a few ways for it to work properly.

Best of luck

0 Kudos
Level 12

I opened a case on this, and SW engineer, Jeremy, gave me a simple fix.

Go to Settings > Manage Accounts

- Edit the account you are using

- Scroll down

- Click the + to expand Voip and Network Quality Manager settings

- Check the settings for IP SLA Operations Summary View

- Set it to "default"

- Click Submit

- Re-test

I noticed my Admin account had them all set to "default", whereas AD groups I had created had the name of the the setting, e.g. IP SLA Web Summary View said IP SLA Web Summary View. I set all of the IP SLA Views to "default".

I check 2 other Orion servers, and they were all set to "default" for the AD groups, so I suspect these got changed when I was troubleshooting a migration problem of IP SLA from one server to another. We had to do some SQL queries to fix some DB errors, and that may have changed the settings, or multiple impatient reinstalls may have affected it.

=Foonly=

Here's how I set mine - I believe this is the default way the installer normally leaves it:

pastedImage_0.png

Thanks for the info, but it looks like all of mine are default, so I don't think this is the issue. We are trying some different things, but may need to log a support case.

0 Kudos
Level 11

I too am experiencing this issue.  Reported jitter and latency values are reversed.

0 Kudos
Level 12

On one of my servers, it looks ok - ICMP Path Echo web page is indeed ICMP Path Echo, and ICMP Path Jitter web page is ICMP Path Jitter.

On another Orion install, the ICMP Path Echo web page has a web page title (that shows up on the FFox tab) that says IP SLA UDP Jitter, the text in the web page says ICMP Path Echo Operation Details.

Clicking Customize Page shows IP SLA UDP Jitter Operation Details View, and the resources would seem to UDP Jitter.

On the same failing Orion install, the ICMP Path Jitter shows IP SLA UDP Jitter in the FFox tab, the page itself says ICMP Path Jitter, and Customize Page shows SLA UDP Jitter.

I tried doing a repair on VNQM with no luck.

So I rolled the dice again, and did an uninstall. I expected the config wizard to pop up, but it didn't. So I did a re-install ... and the config wizard popped up in the middle of that (5-10 minute delay from the uninstall). I guess our system is getting big, but I'll bet other people have stumbled due to the lack of apparent activity.

So I cancelled the config wizard, finished the install (I hate to cancel installs - almost guaranteed corruption), and ran the config wizard manually, then I rebooted.

Still broken.

Any clues would be appreciated, or we'll punt to SolarWinds support within a few days.

=seymour=