Here is the TLDR; for this article. We noticed that the TTFB via the QoE HTTP monitor (you're using DPI via QoE, right? It's free now -- you really should.) spikes up. At the same time we see a corresponding spike in data volume. I need some advice on tracking down the source of the increased volume. Help me Thwack, you are my only hope!
Here are the details:
We are leveraging both QoE (the HTTP monitor) as well as both custom HTTP monitors in SAM and the AppInsight for IIS monitors for our additional web server. As described above, our primary poller sits in what is technically our secondary datacenter, but it is really like we have two primary data centers. The DB server and about half of our APEs sit in the same DC as the primary poller. The additional web server is located in another data center along with the remaining APEs.
Unfortunately, I'm having a tough time tracking down where that spike in traffic might be coming from. There is a negligible increase in traffic on the NIC, no QoE network response time spike, CPU or memory usage spikes. (See the graphs below)
The increased response time can last from 12 hours to a few days and then stops as suddenly as it started. AppInsight for IIS doesn't show a spike in connection attempts, current connections or network IO during the issue -- at least nothing out of the ordinary. Any thoughts on how I can track down the cause of the increased response time? Again, the server itself is not seeing an increase, only the HTTP monitors.