SAM Component monitor not reporting accurate data

Within the last week I created a new I created anew application monitor to monitor a service that's deployed across my companies "server verse". After linking the component monitor to a alert, I got an influx of servers reporting this service was down. This was expected behavior as I suspected there were a few machine that were out of compliance. While working through the alerts I noticed some of the alerts were reporting the monitored service was down but it was in fact up on the node itself. this is obviously a problem because I need to trust this alert. There are also nodes reporting that the service itself is in fact up and running?

Troubleshooting steps:

1. I verified the application template was pointing to the correct service.

2. Confirmed the alert logic was correct.

3. Re-polled the node

3. Restarted the service on the affected node.

Is this a known bug? If so why is only affecting a few nodes.

How can I get this application monitor to poll more accurately?

Find more posts tagged with

application monitor templates

sam

Component Monitoring

Application Component

application_monitor

Accepted answers

All comments

mesverrum

Are you maybe getting timeouts when trying to poll the node?

How do you poll? WMI, SNMP, Agent? Might be something funky going on there.

radswuser

can you post a screenshot of your alert logic/criteria?

dhankerson

@mesverrum Its confusing to me that timeouts would be the issue. All other metrics for the nodes are being captured on the the nodes details page. The nodes are being polled using WMI.

dhankerson

chad.every

Building off of what @mesverrum mentioned, what type of component monitors are you using in your Application Template? Is it the 'Windows Service Monitor'? or another type?

dhankerson

@chad.every I'm using a Windows Service Monitor component type with a WMI fetching method.

chad.every

So I've reviewed your original post and the comments. There are no bugs in monitoring WMI services that I'm aware of and the WMI service monitors have been working perfect for me. I'm thinking we need to better understand the STATUS of the component. Your alert logic only lists 'when status is not equal to up'. It is safe to say that you're probably only looking for a status of down/warning/critical. In reality, there are about 2 dozen different types of statuses in the SolarWinds Platform.

If you edit your alert, navigate to the 'Trigger Action' section and add this text/variable into the 'Message displayed when this alert is triggered'

Component Status: ${N=SwisEntity;M=ComponentAlert.ComponentAvailability}

Save the alert after adding that. It will only show when the alert triggers again, so it wont update on any alerts that are already active. To clear active alerts for this navigate to Alerts & Activity > Alerts. Using the 'Group By option on the left, choose 'Alert Name', select the 'Cortex XDR Service Alert', in the right hand side, select all active alerts and click on the 'Clear triggered instance of alert'

Since you're alert 'condition must exist for 1 hour' is set, it might take up to an hour for the alerts to start rolling back in.

Adding that text/variable above will update this Message section of the alert.

It would be good to know what statuses are being returned for the monitors in question. Int he end, it might mean making adjustments to the alert logic for something other than 'UP'.

adam.beedell

Feels like you've set something up funny. Not sure which thing.

EG, your alert says not up. But you've kept the defaults for CPU/Mem thresholds, which will translate the status into a warning/critical, which will trigger your alert

Alternately, the test goes unknown because creds are negotiated poorly or something, grey status, alert triggers.

Should be able to troubleshoot based on what status they're actually in. Can report tis out fairly quickly, but it'll also be on the SAM Summary dashboard as default