Out of the Box Alerts sanity checking

RichardLetts

can we get someone at solarwinds to check the out of the box alerts for sanity.

e.g. this is a poor alert definition:

All that unmanaging does is stop polling for a specific state. If the state has already occurred then this alert still triggers.

having a packet loss alert on a down node is not helpful, and the alert clears, until you unmanage it, at whch point this alert triggers.

Unmanaged nodes that experienced high packet loss before being unmanaged will continue to generate this alert.

The only way to 'fix' this is to clone the out of the box alerts and alter the conditions.

Node down alerts clear when a node is unmanaged because they check for the status equal to 'down' and unmanaging a node changes its status to 'unmanaged'

Basically all of the alerts which have 'Status is not equal to down' should be switched to say 'status is equal to 'up'

(once a node is responsive i.e. 'up' then polling for other issues will trigger and alerts for components.

adatole

You've hit upon a few key concepts. I want to state that nothing that you said is wrong, AND nothing you've described is broken. The system is behaving as designed AND ALSO it's not what you (the monitoring engineer) need to have happen.

1a) unmanage effectively "freezes" that node in time. No more data is collected, so unless you're alert also considers time  (ie: "started being down in the last 15 minutes") the last status is the last status.

1b) when you re-manage, everything that is happening to the node is happening to the node NOW. SO alerts that tell you that a system is back up, or is now down, or whatever, are going to trigger as if that's been the case all along.

THEREFORE, the "MUTE" option is the thing I've been talking about since literally 2013. Polling continues, but alerting is suppressed.

2a) "where status is not down" is far too broad a category for most alert triggers. You've got unmanaged, unavailable, warning, and of course "up" - all of which fall into the scope of "not down" and thus would trigger this alert. You're far better in most alert definitions to explicitly list (in an OR block) the states you are looking for. Up, plus whatever else fits the bill.

2b) as a nice side-benefit, the query that "is not xx" creates is significantly more expensive (ie: demands more resources) on the database. Given that alert queries run once per minute by default, and organizations can have hundreds of such queries, the aggregate impact of poorly designed alert triggers can significantly impact the overall performance of your Orion installation.

$0.02 happily credited.

- Leon