Alerting Query for one client - suggestions please?

Currently we have dozens of different alerts which are 'shared' across all of our different clients.

Background: So going ultra simple, the node down alert is the same for client A as it is for client Z, etc. We also, [and let me state up front I don't agree with it] have what has been dubbed "intelligent alerting". So, NPM alerts which triggers a batch file under the trigger actions. This batch file generates an event file which is collected by BEM (part of the BMC suite) which in turn routes the file into a One ITSM incident.

The intelligent part is handled by BEM and it essentially silences all alarms for a client for 15m after an alert is triggered.

A scenario could be: an interface down triggers and successfully alerts and generates an incident. For the next 15 minutes BEM logs any other alert for that client internally but 'silently' drops them - as in, no incident is raised. So SolarWinds continues to generate as expected, but our man in the middle doesn't. **

The Requirement: However, one of our internal client teams now has a desire to bypass this and they want an eMail for every*** single alert. I also think this is not the approach to take and we are trying to educate them out of their "we want everything to alert just in case" approach.

The How???:

So this is where you come in - the only method I can currently "see" is to duplicate every alert we have and edit all of them.
- The original set will have additional logic added that basically says 'if = Client Z then ignore'
- whilst the duplicated set will be client Z specific - we can then add a trigger action of 'send an eMail'.

My (current) Answer: So I have recently got in to modern dashboards, and see this as a fantastic approach to re-creating the built-in Top 10 but is client specific. And the opening view is this:

I see this as a reasonable response to their "log everything" request and my "No" response. They can, at a glance see if they have a flood of node downs, etc. This can clearly be expanded on to include other info such as interface downs, nodes in maintenance, nodes muted, etc

So, bottom line, is there any other easy [ish] answer that doesn't involve
duplicating ?40 alerts and editing all of them?

.
.
.
.
.
.

** I have disagreed with this policy from day one on the basis that a simple alert (e,.g. a WAP down) could mask a serious outage (e.g. FW failure or similar) but this change has been driven by management that "just want to see a reduction in incident numbers" - as if reducing numbers fixes problems. Grrrr.

*** I also fundamentally disagree with alert on absolutely everything including items that are managed and monitored by other toolsets.

  • Hi there, 

    This is definitely more of a 'hacky' way to do it, but I would first create a Node Custom Property called 'AlertEmail' or similar. 

    Then you could create an email action on your existing alerts, but with the 'To:' address set to use the custom property you just created: https://support.solarwinds.com/SuccessCenter/s/article/Use-Custom-properties-when-sending-email-alerts

    The final piece of the puzzle would then be to populate the custom property ONLY for customer Z devices, that way the email would only send if the node that triggers it has the email populated and therefore would only send for Customer Z devices. 

    The only consideration for this would be that every time a node alerts that ISN'T a Customer Z device, SolarWinds would output an error that is along the lines of "Invalid Email address" but this would be confined to the ActionsExecutionsAlert.log on the server (The log file could just get quite big is all!)

    Happy to answer any questions you might have, we have a lot of experienced SolarWinds Engineers here at Prosperon that may have resolved something similar too!

    If this helps answer your question please mark my answer as confirmed to help other users, thank you!

    Marlie Fancourt | SolarWinds Pre-Sales Manager

    Prosperon Networks | SolarWinds Partner since 2006

  • Thanks Marlie ...

    I had considered a custom property approach but rejected that on the basis it was over-complicating matters. Plus I'd need to document the what / why & how of it for future users and I'm just not feeling the energy to do that for this specific client team as they constantly demand but never give back.

    Secondly, the fact that errors would be generated is not something I'd considered, and as we have enough of those already I would definitely reject this option.

    Ultimately I'm pushing my modern dashboard idea and hoping this will satisfy their need for information which is up to date and relevant.

    p.s. your reply also explains why one of your colleagues ash just reached out to me on LinkedIn Grin

  • Hi Stuart, 

    Yes I think what you've proposed is a good option. Sometimes we need to work with our clients to not just give them what they ask for, but instead understand where that request is coming from so we can guide them down a better option. 

    The only other thing I'd offer is that the duplicating and editing of alerts for a single custom becomes a lot easier when you have SolarWinds API and scripting knowledge. Something that may take a lot of time by hand can be pretty quickly scripted out, especially when you have a lot of experience like our engineers. 

    If anything like this comes up again, and you feel like you really don't want to go through the manual hassle, it could be very beneficial to have us available to call on - and so it might definitely be worth answering that LinkedIn connection Grinning

    Kind regards, 

    Marlie.