Is it possible to detect and possibly trigger an alert when an alert is edited?

As time goes by, writing alerts becomes more and more complex. Just today I found an alert that was written in such a way that it would trip on zero SAM instances.

This isn't what the alert was originally built to do but due to renaming of SAM instances via the web interface it unexpectedly broke the alert.

Now, I've got code running on another server that evaluates the active alerts and could generate an external alert when an alert ends up in a broken state, but I'd like to keep this inside Orion and write an Orion alert that detects that an alert trigger condition has changed.

Any ideas on how to do this?

Chris

Find more posts tagged with

Accepted answers

All comments

bluefunelemental

I don't have it in front of me but I just built this exact thing.

Essentially I query the DB from a SAM powershell monitor

This exports the query results as a csv file and compares it against the last run export. If there are any differences then it overwrites the saved csv with the new one.

The beauty is an integration with GIT which shows the difference between the two line by line much like NCM does.

Message me offline if you want me to explain in more detail and share the parts. I use it for Solarwinds web.config and other text files as well.

cgregors

I was hoping for something a little more Orion elegant.

What you've described is something I've already done and also taken it to different level.

Take the current trigger query
Strip out the portions of the SQL that are variable (status and such)
Evaluate the stripped query against the Orion database to determine what targets the alert COULD trip against
Look for a difference in the count between the last time I checked and this time.

Example:

Raw Trigger query:

SELECT DISTINCT APM_ApplicationAlertsData.ApplicationID AS NetObjectID, APM_ApplicationAlertsData.Name AS Name

FROM Nodes INNER JOIN APM_ApplicationAlertsData ON (Nodes.NodeID = APM_ApplicationAlertsData.NodeID)

WHERE

(

(APM_ApplicationAlertsData.Name LIKE 'CommVault%') AND

(APM_ApplicationAlertsData.Availability <> 'Up') AND

(APM_ApplicationAlertsData.Availability <> 'Unmanaged') AND

(Nodes.CommVault = 1) AND

(Nodes.Status = '1') AND

(Nodes.Status <> '9') AND

(APM_ApplicationAlertsData.Availability <> 'Unknown')

)

Stripped Trigger query:

SELECT DISTINCT APM_ApplicationAlertsData.ApplicationID AS NetObjectID, APM_ApplicationAlertsData.Name AS Name

FROM Nodes INNER JOIN APM_ApplicationAlertsData ON ( Nodes.NodeID = APM_ApplicationAlertsData.NodeID )

WHERE

(

(APM_ApplicationAlertsData.Name LIKE 'CommVault%') AND

(Nodes.CommVault = 1)

)

Notice how the bold items in the original query have been removed. The stripped query will return ALL the possible application instances that the alert could work against.

BTW: Stripping the SQL became a black art

Bottom line is we both came to a similar solution.

Thanks for the confirmation that there is nothing easier.

Chris

bluefunelemental

‌i didn't explain my solution very well but it can and does compare the trigger query against a past version of itself.

BY all means we can do the same for the results but I thought your question was around the query itself changing?

cgregors

You are correct. My original ask was about the trigger query only. You answer was an extended answer to my question.

I just wanted to share that I've been down that path also and have spent too much time in the dark side of the netperfmond database

Since we're the only ones discussing this we can probably close it off. Someday when Thwack archaeologists find these scribbles, they may look one of us up to explain how we did what we say we did.

Chris

jscheitel

Does the sinister laugh mean you found a way around this? Legitimate workarounds always welcome! :-)