9 Replies Latest reply on Aug 24, 2017 5:53 PM by timt

    Filter: where node is not muted

    timt

      Thank you in advance.

       

      Since the default "Active Alerts View" does not automatically clear the instance of the alert once that node has been set to mute; I have replicated (somewhat) the Active Alerts View to filter in the alerts that needs to be visible to the NOC.  So when the NOC sees the alert, it gets escalated, then the engineer would "mute" the alert, thus, would also remove the alert from view.

       

      In the SWQL code below, when I run the query it takes a loooong time to complete.

       

      Question.  Is there a simple way to filter out if the node has been muted? 

       

       

      Thanks again.

       

       

      SELECT  DISTINCT

       

        o.AlertConfigurations.Name AS [ALERT NAME] 

          ,'/Orion/NetPerfMon/ActiveAlertDetails.aspx?NetObject=AAT:' + ToString(o.AlertObjectID) AS [_LinkFor_ALERT NAME]

      ,Case

      WHEN o.AlertConfigurations.Severity = '0' THEN ('/Orion/images/ActiveAlerts/Serious.png')

      WHEN o.AlertConfigurations.Severity = '1' THEN ('/Orion/images/ActiveAlerts/Warning.png')

      WHEN o.AlertConfigurations.Severity = '2' THEN ('/Orion/images/ActiveAlerts/Critical.png')

      WHEN o.AlertConfigurations.Severity = '3' THEN ('/Orion/images/ActiveAlerts/InformationalAlert.png')

      WHEN o.AlertConfigurations.Severity = '4' THEN ('/Orion/images/ActiveAlerts/Notice.png')

      End as [_IconFor_ALERT NAME]

       

        ,o.AlertActive.TriggeredMessage AS [ALERT MESSAGE] 

        ,o.EntityCaption AS [ALERT OBJECT] 

        ,o.EntityDetailsURL AS [_LinkFor_ALERT OBJECT]

       

      ,CASE WHEN o.AlertActive.TriggeredDateTime IS NULL THEN NULL ELSE (

          TOSTRING(FLOOR(MINUTEDIFF(o.AlertActive.TriggeredDateTime,GETUTCDATE())/60.0)) + 'h ' +

          TOSTRING(MINUTEDIFF(o.AlertActive.TriggeredDateTime,GETUTCDATE())%60) + 'm'

      ) END AS [ACTIVE TIME]

       

          ,o.RelatedNodeCaption AS [RELATED NODE] 

      ,o.RelatedNodeDetailsURL AS [_LinkFor_RELATED NODE] 

       

      FROM Orion.AlertObjects o 

      left join orion.nodes n on n.caption = o.EntityCaption

       

      LEFT OUTER JOIN Orion.AuditingEvents AS AE ON AE.AuditEventMessage LIKE CONCAT('%', CASE

                WHEN EntityUri LIKE 'swis://%/Orion/Orion.Nodes/NodeID=%' AND EntityUri NOT LIKE 'swis://%/Orion/Orion.Nodes/NodeID=%/%'

                   THEN N.NodeName

                          ELSE 'Wrong'

             END, '%') AND [EntityUri] LIKE CONCAT('%=', AE.NetObjectID)

      INNER JOIN Orion.AuditingActionTypes AS AT ON AE.ActionTypeID = AT.ActionTypeID

       

      WHERE o.AlertActive.TriggeredMessage <> '' and o.AlertConfigurations.Name = 'Critical' and [AT].ActionTypeDisplayName not like '%mute%'

       

      ORDER by o.AlertActive.TriggeredDateTime

        • Re: Filter: where node is not muted
          mesverrum

          Muting the source of the alert seems like a really imprecise way of handling this scenario, as muting it would also prevent that node from triggering any other alerts that it should have in the interim.  As an example, node triggers a high CPU alert, your NOC tech escalates and mutes it, the system engineer maybe doesn't prioritize high cpu alerts right away because they are busy with something more pressing, 10 minutes later the node goes down and no alert is triggered because the node is still muted, potentially slipping through the cracks. 

           

          There is already an acknowledge feature that would seem to be the most appropriate way to indicate that an alert has been seen and a ticket opened. This is a version of an active alerts resource that would just exclude alerts once they have been acknowledged.

           

          SELECT

            o.AlertConfigurations.Name AS [ALERT NAME]

            ,'/Orion/NetPerfMon/ActiveAlertDetails.aspx?NetObject=AAT:' + ToString(o.AlertObjectID) AS [_LinkFor_ALERT NAME]

          ,CASE

          WHEN o.AlertConfigurations.Severity = 2 THEN '/Orion/images/ActiveAlerts/Critical.png'

          WHEN o.AlertConfigurations.Severity = 3 THEN '/Orion/images/ActiveAlerts/Serious.png'

          WHEN o.AlertConfigurations.Severity = 1 THEN '/Orion/images/ActiveAlerts/Warning.png'

          WHEN o.AlertConfigurations.Severity = 0 THEN '/Orion/images/ActiveAlerts/InformationalAlert.png'

          WHEN o.AlertConfigurations.Severity = 4 THEN '/Orion/images/ActiveAlerts/Notice.png'

          END AS [_iconfor_ALERT NAME]

            ,o.EntityCaption AS [ALERT OBJECT]

            ,o.EntityDetailsURL AS [_LinkFor_ALERT OBJECT]

          ,case

          WHEN o.RelatedNodeCaption=EntityCaption THEN 'Self'

          When o.RelatedNodeCaption!=EntityCaption THEN RelatedNodeCaption

          End as [RELATED NODE]

            ,o.RelatedNodeDetailsURL AS [_LinkFor_RELATED NODE]

            ,ToLocal(o.AlertActive.TriggeredDateTime) AS [ALERT TRIGGER TIME]

          -- ,o.AlertActive.TriggeredMessage AS [ALERT MESSAGE]

          --,'/Orion/images/StatusIcons/Small-' + n.StatusIcon AS [_IconFor_ALERT OBJECT]

          ,'/Orion/images/StatusIcons/Small-' + p.StatusIcon AS [_IconFor_RELATED NODE]

          ,CASE

          when minutediff(o.AlertActive.TriggeredDateTime,GETUTCDATE())>1440 then (tostring(round(minutediff(o.AlertActive.TriggeredDateTime,GETUTCDATE())/1440.0,1)) + ' Days')

          when minutediff(o.AlertActive.TriggeredDateTime,GETUTCDATE())>60 then (tostring(round(minutediff(o.AlertActive.TriggeredDateTime,GETUTCDATE())/60.0,1)) + ' Hours')

          else (tostring(minutediff(o.AlertActive.TriggeredDateTime,GETUTCDATE())) + ' Minutes')

          end as [Time Active]

          ,aa.AcknowledgedBy

           

          From Orion.AlertActive aa

          join Orion.AlertObjects o on aa.alertobjectid=o.alertobjectid

          LEFT join Orion.Nodes p on p.nodeid=relatednodeid

           

          where aa.AcknowledgedBy is null

           

          ORDER by o.AlertActive.TriggeredDateTime DESC

            • Re: Filter: where node is not muted
              tdanner

              mesverrum is right that you probably want to filter out acknowledged alerts and not based on muting.

               

              But if you do want to filter on muting, the way to do it would be to join Orion.Nodes to Orion.AlertSuppression on Nodes.Uri = AlertSuppression.EntityUri and check that "GETUTCDATE()" is between the from and until datetime values for the alert suppression entry. See https://github.com/solarwinds/OrionSDK/wiki/Alerts#orionalertsuppression

                • Re: Filter: where node is not muted
                  timt

                  Thanks guys for the info.  Keep in mind that this view is only for the "NOC", whereas the "Active Alerts View" will still show other triggered alerts as well.

                   

                  I need to make it where the NOC view (current script) will not display the alert once the node has been muted, sort of like the old ways when a custom property as "mute" is set to "Y", then do not display the alert.

                   

                  Is there something like that with this built-in mute feature?

                   

                  Tdanner, thanks for the tip, but can you please show me how this "GETUTCDATE()" will filter out the muted nodes?\

                   

                  Thanks again.

                    • Re: Filter: where node is not muted
                      tdanner

                      Like this:

                       

                      SELECT  DISTINCT 
                        o.AlertConfigurations.Name AS [ALERT NAME] 
                          ,'/Orion/NetPerfMon/ActiveAlertDetails.aspx?NetObject=AAT:' + ToString(o.AlertObjectID) AS [_LinkFor_ALERT NAME]
                      ,Case
                      WHEN o.AlertConfigurations.Severity = '0' THEN ('/Orion/images/ActiveAlerts/Serious.png')
                      WHEN o.AlertConfigurations.Severity = '1' THEN ('/Orion/images/ActiveAlerts/Warning.png')
                      WHEN o.AlertConfigurations.Severity = '2' THEN ('/Orion/images/ActiveAlerts/Critical.png')
                      WHEN o.AlertConfigurations.Severity = '3' THEN ('/Orion/images/ActiveAlerts/InformationalAlert.png')
                      WHEN o.AlertConfigurations.Severity = '4' THEN ('/Orion/images/ActiveAlerts/Notice.png')
                      End as [_IconFor_ALERT NAME]
                      
                        ,o.AlertActive.TriggeredMessage AS [ALERT MESSAGE] 
                        ,o.EntityCaption AS [ALERT OBJECT] 
                        ,o.EntityDetailsURL AS [_LinkFor_ALERT OBJECT]
                      
                      ,CASE WHEN o.AlertActive.TriggeredDateTime IS NULL THEN NULL ELSE (
                          TOSTRING(FLOOR(MINUTEDIFF(o.AlertActive.TriggeredDateTime,GETUTCDATE())/60.0)) + 'h ' +
                          TOSTRING(MINUTEDIFF(o.AlertActive.TriggeredDateTime,GETUTCDATE())%60) + 'm'
                      ) END AS [ACTIVE TIME]
                      
                          ,o.RelatedNodeCaption AS [RELATED NODE] 
                      ,o.RelatedNodeDetailsURL AS [_LinkFor_RELATED NODE] 
                      
                      FROM Orion.AlertObjects o 
                      left join orion.nodes n on n.caption = o.EntityCaption
                      
                      
                      LEFT JOIN Orion.AlertSuppression Supp ON n.Uri = Supp.EntityUri
                      
                      WHERE o.AlertActive.TriggeredMessage <> '' and o.AlertConfigurations.Name = 'Critical' and [AT].ActionTypeDisplayName not like '%mute%'
                          AND (Supp.ID IS NULL OR (Supp.SuppressFrom > GETUTCDATE() AND (Supp.SuppressUntil IS NULL OR Supp.SuppressUntil > GETUTCDATE())))
                      
                      ORDER by o.AlertActive.TriggeredDateTime
                      1 of 1 people found this helpful
                      • Re: Filter: where node is not muted
                        mesverrum

                        Tim, I think you may be misunderstanding how muting works. If your NOC team mutes a node then no new alerts will trigger from that node at all until the mute is cleared.  Muting isn't intended to hide alerts, it suppresses new alerts from triggering on that node. In the example I gave the Node down alert wouldn't show up anywhere because it will never trigger as long as the node is muted. 

                         

                        Are you planning to implement a workflow where the NOC is regularly muting nodes and then the engineers who resolve the problem have to remember to unmute it when they fix it?  Is there a system for someone forgetting to clear the mute?  I'm just thinking that the work flow you are setting yourself up for is not at all how the mute feature was intended, but does match up exactly to how the acknowledge feature is expected to be used.

                          • Re: Filter: where node is not muted
                            tdanner

                            Your understanding of the intended use of "acknowledge" is correct. Muting is more like a variation on "unmanage". We have no plans to encourage a workflow based on muting nodes in response to alerts - muting is intended for things like maintenance windows where some downtime or service impairment is anticipated. You will generally want to set an appropriate end time on the mute.

                              • Re: Filter: where node is not muted
                                mesverrum

                                Sorry tdanner , meant this toward the other tim in the thread haha, I would be pretty surprised if you weren't already completely aware of the workflows intended by the team at solarwinds when they implement new features.

                                  • Re: Filter: where node is not muted
                                    timt

                                    You're both correct

                                     

                                    For my workflow, I use this script for the NOC views, as it's displayed on the screen with no access to keyboard nor mouse.  Within this NOC View, i added several of the same query, but each one only showing specific Node Down and Application Down from each view.

                                     

                                    This keeps the "other alerts" from out of view to reduce noise and forces the NOC to respond to only down alerts.

                                     

                                    From here if the alert is triggered, the NOC escalates and calls the 'on-call' engineer to look into the issue.  When the engineer logs in, they "mute" the node, thus, removing the alert from the NOC view.

                                     

                                    I also have another custom maintenance view, which shows all the nodes that are muted and by whom, and from the maintenance views, the engineers would 'unmute' when the node is deemed operational again.

                                     

                                    - NOC View

                                     

                                     

                                    Maintenance View