This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Status of multiple services using one monitor

As we all know, ordinarily SAM will create a single monitor for a single service.  If that service is down, the monitor is down.  If it's running, the monitor is up.

I have a situation where a customer wants to monitor two services on the same server, but wants the monitor to only show as down if both services are down.  If either one or the other is up, the monitor should show as up.

I'm trying to do this via a Powershell monitor which assigns a numerical value to the status of each service.  If the service is Stopped, it assigns a value of 0.  If the service is running, it assigns a value of 1.  It then adds the two results together.  If the result is >0 then the monitor should be seen as up.  If it's 0, it's classed as down, and that triggers an alert.

The script I have is this: (apologies if the format jars any veteran PowerShell coders; I'm still a newbie!)

$arrService = Get-Service -ComputerName 'Server1' -DisplayName 'Service1'

$arrService1 = Get-Service -ComputerName 'Server1' -DisplayName 'Service2'

$stat1 = $arrService.status

$stat2 = $arrService1.status

If ($stat1 -eq "Stopped") {$ServStat1 = 0} Else {$ServStat1 = 1}

If ($stat2 -eq "Stopped") {$ServStat2 = 0} Else {$ServStat2 = 1}

$ServTotal = $ServStat1 + $ServStat2

write-host "Statistic.ServTotal : $ServTotal"

write-host "Statistic.ServStat1 : $ServStat1"

write-host "Statistic.ServStat2 : $ServStat2"

Frustratingly, if I run this script through PowerShell on my desktop, it runs correctly; ServStat1 comes back as 1 (because Service1 is running).  ServStat2 comes back as 0 (because Service2 is not running).  ServTotal comes back as 1, and this would show the monitor to be up.  However, if I copy and paste the same script into the monitor, both variables are assigned 1, and the total comes back as 2.  The services are custom, and so don't exist on any other server.  I've tried setting the machine to run from as both my main Orion monitor server, the node that the services are run on, and also by pointing it at another machine in the same subnet as the server with the services on it and I get the same result every time.  I'm not entirely surprised by this given the -ComputerName switch in the Get-Service statements, but at least it eliminates that as a possibility.

I've seen before some .. inconsistencies.. between the flavour of Powershell that runs in Orion and via the desktop, but this one is baffling me.

Can anyone shed any light on what's not right, or perhaps even suggest a more efficient way of achieving the same goal?  In some respects I'd prefer an answer to this quandary as an exercise in PowerShell, apart from anything else!

  • After some jiggery pokery, and fiddling around with different things, it looks as if the problem is the fact that for whatever reason, Powershell is not recognising the text of $stat1 and $stat2 and reverting to what ever is set in the "Else" section of the If function.  I tried reversing the logic, so that it checks to see if the result is "Running", and assign it a value of 1, else set it to 0.  Now I get 0 as my answer.  Unless there is a way of getting SAM to spit out what $stat1 and 2 are being set to, it's a bit difficult to debug!

  • Try placing the application into debug mode, then parse the debug logs located under C:\ProgramData\Solarwinds\Logs\APM\ApplicationLogs. The directory will be named after the Application ID of the applicatio you placed into debug mode, which can be seen in the browser URL when viewing that application in the Orion web interface.

    Debug Application.png

    Application Debug URL.png

    Application Debug Directory.png

  • Is that the entirety of your script? It appears you are missing a proper exit code, which may be contributing to the issue. At the very minimum you would need to add "exit 0;" to the end of that script.

    Scripts Must Report Status Through Exit Codes

    Scripts must report their status by exiting with the appropriate exit code:

    Exit Code

    Meaning

    0

    Up

    1

    Down

    2

    Warning

    3

    Critical

    Any other value

    Unknown

    For example, if you want to inform SolarWinds SAM that a VBScript reports Up status, you would exit the script using code similar to the following, where 0 reports Up:

    Wscript.quit(0)


  • Thanks..I tried that, and it made no difference at all..  I haven't even got as far as reporting whether or not the status is up or not - just clicking "Get Script Output" gives me an incorrect answer.  In addition, I have written a couple of other scripts without that final line and had no problems?

  • I had tried that initially, but couldn't find the logs.  Now I've found them, there are some logs that are showing "Access is Denied" errors when I was trying using "get-wmiobject win32_service" instead of get-service (although obviously the syntax is slightly different).

    Don't see any for the "get-service" version though.

    The Access is Denied error is certainly interesting though.  I'm using the normal service account as a credential, but it does give me another avenue of investigation, so thank you for that!

  • Ok.. things are starting to come together now.  I'm getting "Get-Service : Cannot open Service Control Manager on computer 'Server1'. This operation might require other privileges."

  • It sounds as though you are trying to query a remote server, which should really be done using WinRM. The issue you are likely encountering is that Orion runs under the "Local System" user account, as most/all services do. This account does not have adequate permissions to access resources on remote machines. As a workaround you may want to try using impersonation using a domain admin or equivalent user account, but the proper approach would be utilizing WinRM.

    PowerShell Impersonation.png

  • If your script runs properly on the server that is running the services to be monitored, you can also consider deploying a SAM Agent to avoid having to deal with WinRM issues.

    Be sure to expand the Advanced Properties of the Application Monitor, and change the platform setting to be x64 from x86 if PS installed version is 64 bit

  • When you say "without that final line ", what final line is that? Did you include an exit ?

  • The credential we use for monitoring is attached to the service account which is a Domain Administrator.  I did try the "specify account" setting, but I got the same result.

    I did find through Google'age that the access denied error being shown in the logs is due to a WinRM issue, so how would I go about utilising WinRM in this instance for this monitor?

    *Edit.  Found it.

    Still getting a result of 2 (Meaning that the service status is reporting back as something other than "Stopped", but I've still no clue as to what).  And now there are no errors in the log file.