Monitoring and Alerting - Design Flaw?

APM does monitoring and alerting. One would expect the monitoring to simply collect data and alerting to set the thresholds to kick of whatever alert is needed when the threshold is met. But the APM requires the threshold value to be set in monitoring and the alert simply is based off whether the monitoring succeeded or failed (in general). This seems backwards to me.

Can some one explain the logic of this? am I mistaken in my thinking?

Thanks.

Find more posts tagged with

Alert

monitor

Accepted answers

All comments

RogerWong

I think your way is as equally valid as the current way. It's different, but the result is the same. I'm reminded of Jonathan Swift's Gulliver's Travels: the Lilliputans cracked their eggs on the small end, and the Blefuscans cracked their eggs on the big end.

Bad_Bulldog

I tend to agree with rudethwack on this. The current implementation lacks flexibility since for different thresholds you need different monitors...

Unless... is it possible to trigger the lowest threshold, the one set in the actual monitor, and set higher thresholds in the alerts...

It would work this way:

1. Monitor's threshold is exceeded
2. Firts Alert fires off based on the monitor's threshold
3. Second alert checks for even higher(or lower) figures and fire off if these are encountered

I dived too deeply in the advanced alerts to know if this is possible...

rudethwack

Thanks for the reply.

As a APM user, the current way is not very intuitive to me. Setting threshold values for custom performance counters require writing a custom WMI script with embedded threshold values in the script. This is not a good solution from maintenance point of view.

For example, I want to be notified when "Available MBytes" counter in "Memory" performance object goes to x (some number) or above on machine1 but want to be notified when it goes to y (some other number) or above on machine2. This requires two monitors to be set up one for each machine with x for machine 1 and y for machine 2. What if I want to get notified when counter value reaches z on machine 3? I can not simply assign the existing monitor to machine 3 because the threshold value is different. I need to write another custom script with a different threshold value.

It would have been very simple if I can specify just counter name for a given performance object in monitoring mechanism (without writing any custom script because built-in WMI Performance Counter Monitor type already does this) and then let alert mechanism take care of setting threshold values. This way we do not have to create another monitor and also eliminate the need for a programmer (who needs to understand how WMI works).

Comments?

Bad_Bulldog

Hello,

You can have the WMI throw back the polled value through the use of the WScript.Echo "Statistic:" & intWMIvalue method. In the template for the monitor you can define a warning and critical threshold which can be overridden on a per machine basis.

This should prevent the need for multiple monitors

Bad_Bulldog

For example, here is a script I made to cook the WMI counters to calculate a more precise disk queue using the perfrawdata counters. I throw back the value using the WScript.Echo method I mentioned. I can afterwards specify different thresholds for the monitor on a per computer basis using monitor overrides.

Script:

'=============================================
' Script:  DISK_QUEUE.vbs
' Auteur:  Pierre-Emmanuel Turcotte
' Org.:  CHUQ
' Date:  2009/04/08
' Notes:  Ce script calcule la queue du
'    disque passé en arguments
'=============================================

OPTION EXPLICIT

DIM strDisk
DIM strComputer
DIM objWMIObject
DIM objDiskPerf1
DIM objDiskPerf2
DIM lstArguments
DIM intResult
DIM N0
DIM N1
DIM D0
DIM D1

CONST SUCCESS = 0
CONST FAILURE = 1

SET lstArguments = WScript.Arguments

IF lstArguments.Count = 2 THEN
strComputer = TRIM(lstArguments(0))
strDisk = TRIM(lstArguments(1))
ELSE
WScript.Echo "Message: INVALID PARAMETERS :: CSCRIPT.EXE SCRIPT.VBS [COMPUTER] [DISK:]"
WScript.Echo "Statistic: 10001"
WScript.Quit (FAILURE)
END IF

SET objWMIObject = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")

IF (IsEmpty(objWMIObject) = TRUE) THEN
WScript.Echo "Message: WMI OBJECT NOT INITIALIZED :: " & strComputer
WScript.Echo "Statistic: 20002"
WScript.Quit (FAILURE)
END IF

SET objDiskPerf1 = objWMIObject.Get("Win32_PerfRawData_PerfDisk_LogicalDisk.Name='" & strDisk & "'")
N0 = objDiskPerf1.AvgDiskQueueLength
D0 = objDiskPerf1.TimeStamp_Sys100NS

WScript.Sleep(1000)

SET objDiskPerf2 = objWMIObject.Get("Win32_PerfRawData_PerfDisk_LogicalDisk.Name='" & strDisk & "'")
N1 = objDiskPerf2.AvgDiskQueueLength
D1 = objDiskPerf2.TimeStamp_Sys100NS

intResult = ((N1-N0)/(D1-D0))

IF intResult < 0.1 THEN
intResult = 0
END IF

WScript.Echo "Message: AvgDiskQueueLength = " & intResult
WScript.Echo "Statistic:" & intResult
WScript.Quit(SUCCESS)

josh.clark

Why can't you just create a WMI Performance Counter component that pulls the Available MBytes value. This value is stored as the "statistic" value.

You could then just override the statistic thresholds for those application monitors that need to use a different threshold.

You can also create an "APM: Component" based alert (Type of Property to Monitor = APM: Component). Then just add your alert conditions against the "Statistic Data" field.

-----

The reason we let you specify the threshold on the monitoring side is so you can see a difference when viewing the website. If we only paid attention to this in the alert engine, a process that is running normally would look just like a run-away process consuming 100% CPU. They would both show as up. With thresholds, you can say that a process uses 100% cpu should be in a critical state.

FormerMember

If we didn't allow you to set the threshhold in the monitor, then you'd have to create differnet alerts for different monitors because the values of the statistics pulled back by each monitor can be radically different. You have to set thresholds somewhere.

rudethwack

I have been wondering what Statistic represents. I can go further with your suggestion. Instead of setting Statistic warning and critical threshold values for the monitor definition and overriding on each assigned Node, I can leave the Statisitc fields blank. This way, the monitored data is always in Available status.

The Alert mechanism provides a way to retrieve the Statistic value in the form of "Statistic Data" field (when Type of Property to Monitor is set to APM: Component). I can use this field to retrieve the value of the performance counter (obtained during monitoring phase) and enter a threshold value to check against. This way I can stick with my principle of not setting threshold values on monitors and use them only in alerts. This also eliminates any custom script coding.

I am very greatful for your valuable suggestion.