Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Defining and testing complex alerts

cgregors

I have a request from a application group that wants the following:

On the servers running "someapp"

When a filesystem >= 90% utilization, alert us

When a filesystem with the name "Paging" in the caption has <= 50Mb, alert us

In the case of the server in question the filesystems are captioned:

C:\ Label: 904559dc
\ Label:DATA 20042e63
E:\ Label:Paging 1863a06d

The E: drive is being used as a dedicated pagefile disk.

I have constructed the following trigger condition to solve the request inside one alert.

Question 1: Does this make sense?

Question 2: How can I test the alert (and I already know about the test button)?

I considered the following technique

find the volumes attached to the node in the database with the following sql:

select volumeid, volumedescription, volumepercentused, volumespaceused, volumespaceavailable from volumes where nodeid=2234;

This returns the following:
use the following sql to push the C: VolumePercentUsed over the trigger limit:

update volumes set VolumePercentUsed=99 where volumeid=5627;
Wait for the alert.

Unfortunately the alert doesn't come. So how can I confirm this alert works as I expect?

Chris Gregors

Find more posts tagged with

Accepted answers

sedmo

I agree. The way it reads in the alert configuration interface seems opposite to the SQL it is generating.

The select queries created by the advanced alert configuration app are stored in the TriggerQuery field of the AlertDefinitions table in the database. If you are familiar with SQL it's very helpful to look at the query being generated by the config app in order to troubleshoot problems with alerts.

All comments

mcbridea

Why do you use a none qualifier on the caption contains? Aren't you interested in file systems with "Paging"? I don't get why that line is necessary.

cgregors

The single rule actually is 2 rules.

Rule1 : alert on C: and > 90%

Rule2 : alert on E: < 50Mb free

Since E: contains the pagefile, it typically is > 90% utilization. That is a good thing. When the available space for the pagefile to grow drops below 50Mb, that is bad.

I also don't Rule 1 being applied on E: which (in these systems) always contains the word "Paging" in the caption.

Chris.

cgregors

Why do you use a none qualifier on the caption contains?

An additional reason I did the logic:

Trigger Alert when any of the following apply
Volume Percent Used is greater than or equal to 90
Trigger Alert when none of the following apply
Caption contains Paging

was I could not find any other way of saying this:

Trigger Alert when all of the following apply
Volume Percent Used is greater than or equal to 90
Caption does not contain Paging

Since there is no "does not contain" construct in the alert editor, I had to make it up another way.

Chris

sedmo

I think your criteria would need to look like this.

Trigger alert when any of the following apply
Trigger alert when all of the following apply
System is equal to someapp
Volume Percent Used is greater than 89
Trigger alert when none of the following apply
Caption contains Paging
Trigger alert when all of the following apply
System is equal to someapp
Caption contains Paging
Volume Space Available is less than 52428801

As for testing, it would be easier to modify your criteria to levels that you know will cause them to fire. For instance if you know you have a volume that is 35% used, set your alert to 34% and see if it fires.

mcbridea

Where did you get the "system" qualifier? Is that a custom property?

cgregors

yes, it is a custom property.

cgregors

The trigger conditions work just fine now.

As for the test process, that worked perfectly also. I guess I was trying to solve a simple problem with a complex solution.

Changing the alert thresholds is the best way to go.

Chris.

cgregors

As a continuing discussion of complex alerts, here is another thing I came across.

The first one shown doesn't work, but the second one does. The only difference is putting each "caption starts with" condition into a seperate condition group.

I was getting real annoyed when alerts for excluded filesystems started showing up, and so did the sysadmins.

Broken alert:

Working Alert:

mcbridea

Sedmo,

What do you figure on this one? I'm stumped.

Andy

sedmo

The difference is in the SQL that is created by the two different configurations. The "broken" one creates logic that reads like this:

(paging is equal to standard 5x9
AND
Node Status is equal to Up
AND
Volume Percent Used is greater than or equal to 90
AND
Volume Type is equal to Fixed Disk
AND
ContactName is equal to OSS Unix)

AND

(Caption does not start with /export
OR
Caption does not start with /import
OR
Caption does not start with /oracle
OR
Caption does not start with /opt/oracle)

The "working" one creates logic that reads like this:

(paging is equal to standard 5x9
AND
Node Status is equal to Up
AND
Volume Percent Used is greater than or equal to 90
AND
Volume Type is equal to Fixed Disk
AND
ContactName is equal to OSS Unix
AND
Caption does not start with /export
AND
Caption does not start with /import
AND
Caption does not start with /oracle
AND
Caption does not start with /opt/oracle)

cgregors

I would have expected the second part of the expression to look like:

(Caption does not start with /export
AND
Caption does not start with /import
AND
Caption does not start with /oracle
AND
Caption does not start with /opt/oracle)

since the "Trigger when none of the following apply" was the condition group qualifier. The way I was thinking "none" implies the "AND" of the conditions.