Post-Upgrade - HTTP monitor response time spike

We upgraded from SAM 6.2.2 to SAM 6.3.3 on the weekend -- part of our NPM, SAM, NTA and SRM upgrade. We discovered that bunch of URLs suddenly jumped from a 'normal' response time to excessive response times. It isn't just this single URL but multiple URLs.

Our URL monitors are actually assigned to an external node that we monitor -- a DNS server for our service provider. This ensures that external customer experiences are replicated (sort of) using these URL monitors.

Has anyone else experience a similar spike?

Find more posts tagged with

response time

url

http

Accepted answers

All comments

vinay.by

Hi Josh,

Good day , Howz the Job Engine process running? Do you see a lot of Job Engine V2's running on the poller ?

vinay.by

If my guess is correct, you may want to clear the old queue (Job Engine sdf) files and give it a try again.

jbiggley

Hey vinay.by, long time no chat! I think we've found something. I'm going to post about it in a reply off the main post. Watch for more details shortly.

jbiggley

OK, I think we've stumbled onto something very interesting.

As a recap, we do our URL monitoring by assigning a SAM template with HTTP response monitors to an external node (specifically a DNS server from one of our ISPs). Doing so lets us measure response times from an external point back to our internal network. After the upgrade on the weekend we experienced spikes in response times for a series of components. We were stumped.

We tried changing the browser emulation, tried disabling the following of redirects but nothing seemed to help.

Trying to isolate the issue, we wanted to exclude the network so we fired up a NetPath monitor from the polling engine that the external DNS server (our URLNode) was assigned to out to the IP address of the DNS server. Within a few minutes the count of URL components with critical response times (usually >5K or 10K ms) dropped significantly. At one point it got all the way down to 1 -- 1 lonely URL in our entire environment.

Well that's interesting!

So we started playing. We reverted all of the component changes but left the NetPath monitor in place. Everything stayed pretty much OK.

The chart above shows our playing.

Our first NetPath poll was at 10:38AM. That first point past 10:30 is the poll at 10:40 -- and it was 13ms. The one previous was, at 10:35, was 3,570ms.

At 11:30AM I disable the NetPath monitoring. Poll at 11:30 was 599ms and 11:35 spiked to 24,660ms.

I enabled NetPath monitoring again at 11:38AM.

I don't know how to explain it, but it seems that enabling NetPath monitoring to the target URLNode (the node that our URL components are assigned) IMPROVES the response times in the post-upgrade world to levels that we would expect to see from the pre-upgrade environment.

Has anyone else experienced a similar issue? I am going to get a case opened up and escalated to development because this is definitely not normal as far as I can tell.

PS: I am going to disable the Netpath probe this afternoon to see if the problem returns. I'll leave it disabled for a little while to see just how bad things can get and then we re-enable.

aLTeReGo cobrien

aLTeReGo

The issue sounds like it's more likely related to DNS name resolution than the actual performance of the website itself. SAM's HTTP/HTTPS User Experience Monitors measure the total experience time, inclusive of all aspects of the connection. This includes resolving the DNS hostname in the URL provided, no differently than an end users web browser would when accessing the site. The likely reason why you are seeing different behavior when NetPath is enabled vs. when it is not, is due to DNS cache. NetPath is very likely keeping the DNS record alive on the server, whereas SAM's default polling rate allows enough time for the DNS record to expire out of local cache and require the lookup process to occur again.

This is just a theory, but a reasonable one given the information you've provided.

jbiggley

adatole and I had a similar discussion about arp cache but I think the same principle applies, but the issue is that our NetPath probe is only pointed at the upstream DNS server and was set to every 5 minutes. The HTTP component monitors are also set to poll every 5 minutes. We have 1387 URL component monitors all assigned that upstream DNS server. I would expect that there are multiple queries from our polling engine to the DNS node every second in the 300 seconds between polling intervals -- upwards of 4+ URLs being checked every second.

I am completely game with the idea that a DNS cache is part of the problem but the question is "Why did this only appear after we upgraded to SAM 6.3.3?" and "Why does Netpath help mitigate the issue?"

jeilers

Shot in the dark but was there ever an answer to this question?
Seeing some similar behavior in my environment.
SAM Version: 6.3.0
Not quite 6.3.3 but our off site web monitor is not seeing the response time like our SAM http/s monitors.

Getting up to 3000ms just about every other time on the SAM Monitor and the offsite is rarely reaching that height. Turned on debug logging and was not seeing errors. Just a high response time.

Any insight would be greatly appreciated, thank you.