trigger actions missing: how to track and restore?

Noticed that in a bunch of OOTB alerts (application, components, and probably a whole lot more) there are no OOTB trigger actions. Reset actions are there, trigger actions - MIA (haha).

So we are getting swamped by OOTB "reset" emails from the usual suspects (SQL, IIS AppInsights, some other OOTB alerts generating whole lots of false positives) - but getting no alerts on those - or much else for that matter.

E.g. the "trigger actions" tab on one of the most prolific OOTB alerts: no actions, not even NetPerfMon.

The "reset actions" tab: OOTB actions are present.

The actions aren't there in "action manager" either - I've looked.

I know I didn't do it - would normally disable an alert rather than deleting an alert action. I am guessing someone got annoyed by them and just deleted them... Or more likely, they either were not there from the get go, or got destroyed during an upgrade. After all, the Orion instance was pristinely fresh (if a couple years old) when I showed up: the SMTP server wasn't even set yet, nor the default email addresses.

Questions:

Possible to track down who deleted them - or at least when? Could this be hiding somewhere in audit events?
What's the easiest way to restore them? (Don't see a way to clone an alert action - unlike an alert itself.)

Thanks!

P.S. For a bit of context: this could be a "pre-existing" condition, hard to say. When I joined my current job earlier this year, the SolarWinds instance wasn't active and seemed largely untouched. Either it was "touched" (alert actions deleted) - or someone else did this recently. If the latter, I'd love to be able to tell what happened and why.

P.P.S. "SolarWinds Platform, NTA, NPM, SAM: 2023.1.0", is what we're running.

Find more posts tagged with

orion

trigger_actions

Alerts

Accepted answers

All comments

monitoringlife

For your last part about auditing changes, there was a recent post that had had to query out of message center and a query for a custom dashboard.

akhasheni

Thanks! Don't see much there (in the audit log). Added an alert trigger action to an existing alert, saved, checked the audit log:

Alert integration state for AlertID '12' was set to 'Disabled' by (me)

Don't even know what it means. I didn't disable anything - only added a trigger action.

Searching these "Alert integration state" events to the beginning of time (1-Jan-1900 seemed a good approximation): nothing suspicious, don't see anyone doing a bunch of these outside of "system" enabling all alerts when first configured, and then me tinkering with them:

Alert integration state for AlertID '' was set to 'Enabled' by 'SYSTEM'.

Just for kicks, added another trigger action to an alert, saved, then deleted and saved. This is all I am seeing in the audit log:

Is this a bug? A feature?

Come to think of it, deleting a trigger action can have consequences as severe as disabling or deleting a critical alert. I'd really like to see at least "an alert has been updated" - or "trigger action deleted, added or changed" type of an event. Doesn't seem like it's an option.

monitoringlife

Only thing I can think of it saying disabled right after creation is if your doing 'execute an external program' you have to manually approve/enable those now for security (after the 2023.2 versions)

Approve alert actions executing a script

akhasheni

Mine is 2023.1.0, still, and I see nothing needing approval.

... and there are no scripts in those actions. Just the usual "send an email" and "NetPerfMon log" ones.

bobmarley

Thanks for mentioning this. I just schedule our upgrade to 2023.3 - three weeks out. Have not read all of the release notes yet.

stevenstadel

For 2. What's the easiest way to restore them? (Don't see a way to clone an alert action - unlike an alert itself.) - The easiest way to clone an action would be to click Assign Action, select one of the actions you want to copy, then Copy and rename it.

akhasheni

...except there isn't an action to copy or assign if the original action (that was specifically designed for a specific alert type - like "component" or "application") was deleted and is MIA?

I mean, sure, I can just re-create it modeling after an existing alert - or better yet - manual copy-paste from the nearly identical "reset" action - which luckily is still there... Was hoping for something a little less labor-intensive... Like a magic "restore default trigger actions for all OOTB alerts where those actions are missing" button... (One can dream...)

(Thanks for trying to help!)

stevenstadel

The OOTB alerts and actions may get recreated by running the configuration wizard. It could be an option to try.

adam.beedell

Much like the others reply I think I can only offer partial help on a couple bits

1) In the magic restore button - > Database restore to temp location from earlier backup, copy just some stuff from the triggeractions (sp?) table

2) Failing that the Copy or Assign stuff - If you've got a source of an action, you can assign that to a different alert through the Actions tab. It could be possible to do something like Find new copy of these alerts. Duplicate alert. Assign action from deuplicate, delete duplicate.

Could get duplicates from config wizard or a trial install perhaps.