Maybe I'm overthinking this, but I'm really struggling with what should be a relatively simple (I think) alert.
I have a couple of VMs that hang periodically; they still respond go ICMP, but not anything else, so they are up, but essentially useless. I figured this would be a good time to test an alert to auto-reboot a VM. I have an existing query that identifies the VMs, since they stop responding to statistics polling. I have based the report on the LastSystemUpTimePollUtc column in Orion.Nodes, basically looking for any nodes that haven't updated this timestamp for more than an hour.
I am using these specific nodes in an alert with a custom trigger condition, which looks like the following:
SELECT Nodes.Uri, Nodes.DisplayName FROM Orion.Nodes AS Nodes
WHERE (Nodes.Caption = 'server1' OR Nodes.Caption = 'server2') AND HOURDIFF(Nodes.LastSystemUpTimePollUtc,GETUTCDATE()) > 0
I have, of course, replaced the actual server names in this example, but I think you get the idea.
I have tested the query. Both of my problem VMs are in this state at the moment, helpfully, and the query reliably returns two records which, I assume would trigger a separate alert for each node.
For my reset logic I thought the following would be reasonble:
SELECT Nodes.Uri, Nodes.DisplayName FROM Orion.Nodes AS Nodes
WHERE (Nodes.Caption = 'server1' OR Nodes.Caption = 'server2') AND HOURDIFF(Nodes.LastSystemUpTimePollUtc,GETUTCDATE()) < 1
with the idea of testing for a successful poll sometime within an hour period. However, I'm a little unsure of how the alert engine deals with this, since this statement would be true if EITHER of the servers in question polled successfully. There is nothing here to indicate that I only want to compare this value on the node that triggered the alert.
Maybe I'm approaching this all wrong, but I figured this would be the right place to ask. I have read through Richard Letts great post in a similar vein (Warning about custom SQL alerts (reset trigger)), but it doesn't exactly answer my question.
I would appreciate any thoughts you may have. Thanks!