just a "stab in the dark", but what is the utilization durring those peaks... and or QOS?
We do not use QoS.
Do you mean utilization on the outside interfaces? It is very small.
BIG ASA tunnel(outside)...to..................SMALL ASA or PIX tunnel(outside)
(100 Mbps ethernet)........to...................ISP Cable-modem or DSL
(utilization < 10%.............to..................utilization = unknown)
First off, "response time" of what?
Let me demonstrate a critical difference. If I'm monitoring a Webpage in that remote site I'm getting a measure that is directly related to end-user/service experience. If I am monitoring a switch/router via PING & SNMP, my response time is a measure of the mgmnt capability of that device to respond to mgmnt requests, and not a measure of the traffic passing through the hardware.
Ever log into an older Cisco switch, especially 4000 series with CAT OS? You do a SHOW CONFIG and the first page show up, then it halts, possibly for over a minute before it will then continue. Every user on that switch didn't just freeze. Mgmnt functions operate at a separate priority and while ping and SNMP may be slow as dirt, traffic is flowing because of all the hardware-based functionality. Functions that aren't handled by chips, but by CPU and software processing are much slower.
So we regularly see variances in response time to Solarwind's SNMP & Ping requests that do not directly reflect user experience. They can be an indicator of a load on the device--but that load may be about config and not what users are doing. For instance, someone created an ACL with too many "LOG" entries. Every packet that matches the LOG line punts to the CPU for software processing and drives up the CPU, slowing the device with too much management traffic.
Second, your endpoint's utlization/performance is an unknown and that leaves a big hole in any conclusions you are trying to reach. We have a VPN from NY to Rome, and our ability to get responses from Rome falter when Rome's pipe gets heavily used or when NY is heavy. International performance has vastly improved from the 90s when a ping to websites India was over 1500 msec. But every hop is a opportunity for variance. Also I've seen heavy BGP route changes in the last year amongst ISPs as they constantly jockey traffic to lowest cost paths. DSL options can often carry zero assurances and can be slushy with performance.
Lastly, I've seen subtle issues on routers with certain IOS features (FW/Inpsect in particular). They would cause performance problems, often random that were difficult to nail down. They don't always make themselves obvious with 100% CPU or error messages that lead you down a clear path. Those plainly ****.