Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

How Can I produce an alert when the Response Time of a node is X times greater than its average?

I found a wounded server today that had been v-motioned onto an ESXi host that was having resource allocation issues. By wounded I mean it was still green in SW but, it was very slow and sluggish. The Windows VM showed 62% CPU used and 20% of its memory still available. I wasn't getting much out my VMAN integration other than there were several VM guest on the host in a critical state. I don't alert on that because I do all the Windows monitoring, and I find alerting on host is critical to produce many false positives. That said, I did notice that VM response time over the period of the issue was significantly higher than its average. How would you go about producing the alert I have described in the subject? Sounds like Custom SWQL or SQL to me. I'm not finding anything OOB as a starting point that doesn't use QoE. Maybe Something like this?

SELECT Nodes.SysName, Nodes.DisplayName, Nodes.[AvgResponseTime], Nodes.[ResponseTime], Nodes.[AvgResponseTime]*10 AS theResult

FROM Orion.Nodes AS Nodes

WHERE Nodes.[Status] = '1'

AND Nodes.[AvgResponseTime] > 0

AND (Nodes.[ResponseTime]) > Nodes.[AvgResponseTime]*10

Find more posts tagged with

Accepted answers

All comments

JP_net

What modules do you have access to? Do you have VMAN or SAM? With VMAN you should be able to see resource problems on you hosts. VMAN also contains Recommendations, Datastore metrics and others that our clinets have found to be of use in scenarios like this. SAM could help you deploy possible HTTPS monitors to your web servers to measure performance. Also I would assume in this scenario that any SAM components that you would apply to these servers would be able in critical or down state if this were to be happening.

This could be a simple response time alert. However, I would be careful with that. When you configure a response time alert you might get one of those every time that something goes down. That might get annoying and cause some alert fatigue with you and your colleagues.

jameslindsay

HCO - Enterprise plus SRM and WPM. My attention was brought to the issue by a failed WPM Transaction and after snooping around I found many SAM templates in an unknown state. Looking at VMAN, other than the VMs showing CRITICAL, the host appeared to have no issues. I do wish VMAN would be improved to pull in alerts from vRops. I'm also pulling ESXi host info via SNMP.

JP_net

SAM components in unknown status is a great alert not to be overlooked. Sometimes when changes happen in environments your templates will go into this state because they don't have correct permissions. That alert would capture that event as well. If you have anything going into unknown state, you aren't monitoring it.

Having said all this, really understanding server issues where the OS is half down and half up can be tricky. The next time something like this happens it could present itself differently.

I feel like you are on the right track with getting SNMP on your hosts. Consider though that VMAN is capturing a lot of what is there already. What do you have for VM based alerts? Hosts in critical state, CPU, Memory, datastore alerts could be useful here.

One other thing that might not be related to this problem, but could be. Make sure you are getting in your iDRAC, ILO or equivalent as SNMP nodes too. This can net you a lot of your hardware statistics like physical hard drives, power supplies, fans etc.

adam.beedell

Your SWQL looks about right, except you probably want to set avg response time over a specific window using either a separate element or a custom property, the one used there is over a short range I believe.

I'm short a test env for a few days-

Downsample(d, p) - Rounds the supplied timestamp d to the defined time period p. For example, a period of '00:15:00' would round to the nearest 15 minute time increment. Requires Orion 2018.3 or later.
Avg(n) - Returns the average (arithmetic mean) of the values in the group
GROUP BY
WHERE Timestamp > ADDDAY(-7, GETUTCDATE())

those functions would produce the averages

jameslindsay

Thank you for that tip. I was trying to figure out what SW was doing as it relates to calculating Nodes.[AvgResponseTime] wondering how many datapoints over what period of time and does it exclude the lowest and highest value of the dataset, etc.

jameslindsay

mesverrum

I havent tested that attribute specifically, but my gut guess would be that avgresponsetime might be the rolling average across the last 10 polls, similar to how they do percentloss. It's also possible that it is just the latest response time value straight up with no math, there are plenty of places in the DB where they show min/max/avg in the column name but dont actually do any math until that data gets rolled up in a week or month.

Generally if i want something historical I do my own join to the responsetimehistory table and then math it up however i prefer from the raw data.