Dynamic group assignment based on polling engine ID

I am working on a form of automatic fail over in the event a polling engine fails.

Seems like an alert could be set up with an action:

Alert:

node selection: (all nodes assigned to engine 5)

Alert Trigger: (engine 5 services, interface, (whatever you want) not working

alert action: execute (script action, sql update, best solution) to move effected nodes to engine 11, only have to change one value in the DB.

Here is what I have now: alert suppression using dependencies:

Custom property_poller: value = poller1, poller2, poller3, etc....

Group with dynamic query: if CP_poller = poller 1, assign to group 1, (etc..)

Depenency: all members of group 1 are dependant on poller1 interface being operational.

This solves the problem of sending off 1000 alerts, when someone accidentally disables the interface, but I think the ability to assign members to a group based on the value of their engineID would be more direct and more efficient.

Just need a way to use engineID in a dynamic query, and use engineid as the node selection criteria in an alert that can execute the action to change the engine id on the nodes.

For example:

Group1 = all nodes assigned to poller 1.

group1 is dependant on poller1 being operational

alert:

all nodes on poller1.

if poller1 is down, execute action to move nodes to poller07.

this way once the nodes are moved they are also moved to group 7 and alerting will automatically resume, but will not alert while poller1 is down and they are assigned to poller1.

So the question is, is it possible to assign nodes to a group based on actual assigned polling engine?

Can an alert object selection criteria be selected based on assigned polling engine?

Can a Sql update or something similar be created as an action?

Find more posts tagged with

polling engine

custom alert

disaster recovery

action script

trigger action

fail_over_engine

dynamic query

pollers

Accepted answers

mesverrum

Taking those questions one step at a time,

So the question is, is it possible to assign nodes to a group based on actual assigned polling engine?

The groups dynamic query UI does not have an option in their pull down for engineid, but what you can do is set up a node alert that is always true (example nodeid is not null) and as the action it can set the CP_Poller property to be "Poller ${N=SwisEntity;M=EngineID}" then keep your existing group dynamic query where poller1 goes in group 1

Can an alert object selection criteria be selected based on assigned polling engine?

Yes, but in this case it probably isn't relevant to what you are doing. To kick off the action you are looking at I would probably just have an alert looking at polling engines and if any of them have greater than 5 minutes since keepalive then it triggers the action below.

Can a Sql update or something similar be created as an action?

Not directly, but you can set up an action to run an external program. What I would probably do in this case would probably be to launch a sqlcmd script that accepts the triggering engineid as an argument and updates the engineid of all the nodes where the engineid = triggering engineid to the next one in a list. You could also use powershell or vbscript or whatever you are most comfortable with.

-Marc Netterfield

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems

All comments

mesverrum

Taking those questions one step at a time,

So the question is, is it possible to assign nodes to a group based on actual assigned polling engine?

Can an alert object selection criteria be selected based on assigned polling engine?

Can a Sql update or something similar be created as an action?

-Marc Netterfield

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems

jacob.johnson

The solution you provided is very great.

I have implemented the first part and will work on the sqlcmd script later, if needed.

however there is an issue. I did everything exactly as you stated. The alert to set the custom property to the polling engine id is the issue.

it worked, I set it to run every 2 hours, and initially did an 11 second reset, this was not enough time to update all the values and it did not finish. Changing the reset to 15 minutes resolved the issue and all fields are updated.

The problem is, this is going to clutter up the active alerts view for quite some time, and over and over again. I changed the alert condition to Custom_poller CP is not equal to ${N=SwisEntity;M=EngineID}, but only 252 nodes were found to be vulnerable, at the time it should have been over 3000 nodes. the question is if I do in the alert trigger CP_poller does not equal ${N=SwisEntity;M=EngineID}, is the alerting engine grabbing the engineid of one nodeid or something different at that stage in the process. and is there another way to set up the alert to only trigger and change the cp_poller value if the cp_poller value and the engine id in the DB do not match?

jacob.johnson

Another update.

The Custom property I created is Text. all the engineids did automatically assign to the custom property, but when I go in the db and do

SELECT c.CP_poller, n.engineid, n.caption

FROM nodescustomproperties c

Join nodesdata n on c.nodeid=n.nodeid

WHERE polling_engine_id != 'n.engineid'

I get every node returned. so at the very base level the 2 values do not match according to the query, do I need to create another cp and use the type integer for the comparison to even work at all?

jacob.johnson

okay, because of your wonderful guidance I was able to figure out the alert piece, in case you are interested:

I just used the double value trigger option and put cp_poller is not equal to EngineID.

It worked like a champ, I changed several cp_poller properties and moved several nodes to different pollers and everything automatically updated.

just think, after I get the automatic moving of nodes to work, I can essentially just turn off a poller for several days, barely miss a single polling cycle, and no extra alerts will be sent out, this is going to be awesome, thanks to your out of the box thinking and correlation between system abilities and requirements.

mesverrum

Glad to know you were able to work it out, I've been in airports all day today so I wasn't able to check back in until now.

Definitely let me know if you get stuck on anything else, but it sounds like you have a good eye for solving your problems.

-Marc Netterfield

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems