SAM: Script or other mechanism to auto-mute alerts on a user initiated server restart?

I'm checking to see if anyone has done anything creative on this scenario. Frequently we have server owners reboot servers and create a false positive "server down" alert. I know one option is to extend the time SW checks the servers online, but is there a way from either the windows server side or the SW side to mute an alert if Event ID 1074 is encountered: "Event ID 1074: This event is logged when an application is responsible for the system shutdown or restart. It also indicates when a user restarted or shut down the system by using the Start menu or by pressing Ctrl+Alt+Del.

I know there's lots of process things that could get around this but they're humans and it continues to happen. Trying to merely curb "cry wolf" alerts that tend to get ignored. Especially when our server down alerts are integrated with our 24/7 NOC.

Thanks in advance for sharing any experiences, thoughts.

Find more posts tagged with

sam alerting

sam

Alerts

server and application monitor

Accepted answers

All comments

SteveK

Unfortunatly not that I'm aware of. Keep in mind that SolarWinds does more of a moment in time monitor than real time. So for SW to catch event id 1074 before the reboot would be next to impossible. My advice would be the threshold like you mentioned, maybe have that set to 5 minutes. That way a server has to miss 2 cycles before it reboots. If it takes longer for the system to reboot, then that goes into the balance of whats the appropriate threshold for a server to be down before notifying.

matta785

Hey thanks SteveK. I couldn't think of a way either but its reassuring what you say is what the path i've been on in the past. Appreciate ya.

SteveK

Only other thing I could think of would require creating additional alerts. Have one alert for normal business hours that would have a tier2 escalation after X amount of time. Tier1 emails the internal team(s) then Tier2 hits the noc. Then an alert for outside those hours that immediately goes to the noc. That one goes more into what time of day the people are rebooting servers.

Depending on your setup though, the amount of alerts could get a bit unmanageable. We ended up using custom properties to determine what emails or pages, as well as a field for email address so we didn't have to create additional alerts for individual teams.

donrobert5

I think you can add a trigger wherein node name is not equal to (Node Name of Window server) or add a custom property with windows server in it and exclude that and create a new node down alert with 5 minute delay which would be equals to the window server. You should have to be creative.

bobmarley

I would tend to agree with @SteveK If you absolutely wanted to unmanage or mute the node for Event ID 1074 you could do it by using an alert action to execute and external program, passing the node information along variable and creating a powershell script to mute or unmanage the node for X minutes

adam.beedell

Ignoring the "get em to mute", 3 approaches

1a) apply that sam template to everything
b) node down alert includes a phrase about AND that application is UP
c) Note that that is reasoably easy in SWQL but not in the UI
d) Double note - You're going to have the "down" information before you get the 4071 information by at least 1 poll, so it wont work without

2) Add 1 poll cycle worth of smoothing

3) Attach an external program action to the alert, feed the program the nodeID or URI. Program should accept that as an argument and mute the ID given for ~30 mins or whatever. This is easier than it sounds but is a bit fiddily.