Open for Voting

If...Then...Else statements for Alert Manager

I think it would make managing alerts much easier if we were able to build an if-then-else statement into the actions for an alert.

My example...

I create an alert for a particular set of criteria, but I want to enable a different response depending on who is responsible for solving the issue.

IF Node.Owner = "Server" THEN do(Action1)

IF Node.Owner = "Network" THEN do(Action2)

I gather it would be fairly complicated to implement, but I think it would really improve the alert manager.

Parents
  • Up voted but you can already do this with SQL macros.

    For example:

    ${SQL: SELECT CASE

    WHEN NodeName='Server123'

    THEN 'blah'

    WHEN NodeName='ServerABC'

    THEN 'whatever'

    ELSE NULL END AS Foo FROM Tablename}

  • We already use macros to change the way alert actions work, but as far as I'm aware, we can't use them to select different actions depending on properties. But I'm by no means an expert.

    For example if we wanted to something like "if server priority =1 make automated phone call else send email".

    Currently, we do the above by having separate alerts configured for P1 systems. 
    If this can be done with macros, I for one would be interested to learn how. 

  • At the heart of what we want is really an alert workflow where you can have logic define different actions.

    You can kind of that do that with SQL macros albeit somewhat limited. For example if you wanted to send an email action to different SMTP recipients based upon say node custom property that represented a priority you could use this SQL macro in the To field of the email action. It doesn't appear that you can add variables to the TO field but you can.

    Something like this would evaluate a custom property on a node and output a different SMTP recipient based upon the value of that Node that triggered the alert.

    ${SQL: SELECT CASE

    WHEN Nodes.CustomPropertyName='High'

    THEN 'P1_DistributionList@company.com'

    WHEN Nodes.CustomPropertyName='Medium'

    THEN 'P2_DistributionList@company.com'

    WHEN Nodes.CustomPropertyName='Low'

    THEN 'P3_DistributionList@company.com'

    ELSE 'CatchAll_DL@company.com' END AS Foo FROM Nodes

    WHERE ${NodeName}=Nodes.Caption}

    Of course this would always fire the same alert action for each condition and that may work for you but if you wanted to alter the content of the alert action you could nest SQL macros inside the body of the alert action to alert the content too.

    I use SQL macros mostly to apply logic to my restart actions (ExecuteExternalProgram). For instance if a SAM HTTP/HTTPS component fails I will have it restart IIS or Apache using this logic.

Comment
  • At the heart of what we want is really an alert workflow where you can have logic define different actions.

    You can kind of that do that with SQL macros albeit somewhat limited. For example if you wanted to send an email action to different SMTP recipients based upon say node custom property that represented a priority you could use this SQL macro in the To field of the email action. It doesn't appear that you can add variables to the TO field but you can.

    Something like this would evaluate a custom property on a node and output a different SMTP recipient based upon the value of that Node that triggered the alert.

    ${SQL: SELECT CASE

    WHEN Nodes.CustomPropertyName='High'

    THEN 'P1_DistributionList@company.com'

    WHEN Nodes.CustomPropertyName='Medium'

    THEN 'P2_DistributionList@company.com'

    WHEN Nodes.CustomPropertyName='Low'

    THEN 'P3_DistributionList@company.com'

    ELSE 'CatchAll_DL@company.com' END AS Foo FROM Nodes

    WHERE ${NodeName}=Nodes.Caption}

    Of course this would always fire the same alert action for each condition and that may work for you but if you wanted to alter the content of the alert action you could nest SQL macros inside the body of the alert action to alert the content too.

    I use SQL macros mostly to apply logic to my restart actions (ExecuteExternalProgram). For instance if a SAM HTTP/HTTPS component fails I will have it restart IIS or Apache using this logic.

Children
  • Yes, we already use code similar to this to steer emails to the right recipients.
    What we can't currently do though is choose different trigger actions depending on values. e.g.

    WHEN Nodes.CustomPropertyName='High'

    THEN run a program that makes a phone call (e.g. Xmatters)

    WHEN Nodes.CustomPropertyName='Medium'

    THEN run a program that sends an sms

    WHEN Nodes.CustomPropertyName='Low'

    THEN send an email