10 Replies Latest reply: Jul 6, 2015 11:08 AM by avrilspirit RSS

    Alert / Event from Syslog or Trap

    poindexter

      Hello thwack community,

       

      Having read what I could find about the subject, I am in despair to find a way to produce visible, interpreted content in the Alert or NetPerfMon Event log in response to an SNMP Trap or Syslog entry.

       

      The actions afforded in the Trap/Syslog Rule Alert Actions are divorced from the actions available in the Advanced Alert manager.  I am unable to set an alert in the way I can from a polled element.  I would like this capacity.

       

      The options seem to be:

      1) Create a rapid Universal Poller for the values that the traps are reporting on

           This does nothing for Syslog

      2) Launch an external program / script that can insert Alerts and / or Event entries

       

      Option 1 is pretty silly, generating a lot of traffic and introducing delay in response.

       

      Option 2 seems to be the way to do it, but the Orion SDK documentation does not make it clear what is an is not accessible through a CRUD/SWIS operation.  Here are some potential strategies:

       

      2a) Do a managed (via some SWIS API call) triggering of an alert.

      2b) Do a direct (via SWQL) insertion of an Alert or Event.  Has this been done?  Is it possible?

      2c) Set a custom property on a node indicating the current error

           This would require a universal poller for the Advanced Alert Manager to pick up... with a high polling rate. =\

       

      Any help would be appreciated.

       

      P.S.: Does anyone else find it a little ridiculous that SNMP traps and Syslog entries are not able to trigger alerts out of the box?

        • Re: Alert / Event from Syslog or Trap
          RichardLetts

          I've posted on this topic extensively, and there are several ideas out there for improving the alert engine. the Trap and syslog processors work in real-time on incoming messages, whereas the alert manager runs asynchronously based on what is in the database. You have to think hard about how this is going to work together (if it were easy SW would already have fixed it)

           

          Right now I think the best solution is to use custom SQL alerts.

           

          syslog:

           

          in the syslog engine I match messages and set the messagetype; this allows me to flag similar syslog messages with the same message type. Unfortunately you can't export these or I'd include an example.

          Match: *UPS: An inverter fault exists. 0x0165*,*UPS: In bypass in response to an internal hardware fault. 0x011B*

          Modify the syslog, set the message type = 'UPS_ERROR_INTERNAL',

           

          the advanced alert rule then becomes a custom SQL alert on a node with the following condition

          where nodeid in (
          SELECT nodeid
          FROM   syslog
          WHERE  messagetype ='UPS_ERROR_INTERNAL'
                 AND datetime > Dateadd(day, -1, Getdate())
          )
          
          

           

          and the reset condition is:

          where nodeid NOT in (
          SELECT nodeid
          FROM   syslog
          WHERE  messagetype ='UPS_ERROR_INTERNAL'
                 AND datetime > Dateadd(day, -1, Getdate())
          )
          
          

           

          For traps I do a similar thing:

          match trap conditions, set a TAG in the trap rule, and then use a custom SQL alert like the following:

           

          WHERE  nodeid IN (SELECT nodeid
                            FROM   traps
                            WHERE  datetime>Dateadd(hour,-1, Sysdatetime())
                            and tag='OSPF_IF_CONFIG_ERROR'
                           )
          
          

           

          /RjL

            • Re: Alert / Event from Syslog or Trap
              poindexter

              Yes, I've read your posts in regards to this matter, and you're advocating option 2c here.

               

              It's not ideal.  There is data within the trap that I would need to pass to the alert, and it doesn't seem like the Tag action has the ability to extract information from the trap.

               

              Do you have any experience using the "SDK"?

                • Re: Alert / Event from Syslog or Trap
                  RichardLetts

                  no, I'm not advocating option 2c -- I'm advocating the option you don't list. The syslog and trap receivers process the values as they come in. You don't need a universal device poller at all, you can sumply do a custom SQL alert in the alert manager.

                   

                  You can tag the trap with anything that it has extracted from the trap, e.g. ${vbData5} for the 5th varbind in the trap.

                   

                  I use the SDK extensively; I don't see that it will help you here.

                   

                  Why not try my suggestion and share where you get stuck.

                   

                  /RjL

                  • Re: Alert / Event from Syslog or Trap
                    RichardLetts

                    oh, if you need specific things from a trap then you can do something like this in the alert advanced alert actions

                    write a message to the event log with the following text :

                    PBX alert on node: ${Caption} ${SQL:SELECT top 1 tv1.oidvalue +' - '+ tv2.oidvalue + ' - ' + tv3.oidvalue + ' - ' + Substring(t1.tag, 1, 20) FROM traps T1 left outer join traps t2 on t1.nodeid=t2.nodeid and t2.colorcode=65280 and Substring(t1.tag, 1, 20)=Substring(t2.tag, 1, 20) inner join trapvarbinds tv1 on t1.trapid=tv1.trapid and tv1.oidName='g3alarmsAlarmType' inner join trapvarbinds tv2 on t1.trapid=tv2.trapid and tv2.oidName='g3alarmsMaintName' inner join trapvarbinds tv3 on t1.trapid=tv3.trapid and tv3.oidName='g3alarmsPort' WHERE t1.nodeid = ${NodeID} and t1.Colorcode= 65535 AND t1.datetime >= Dateadd(hour, -4, Getdate()) and t2.trapid is null  }

                    (which extracts the last non-cleared Avaya PBX alert type from the trapvarbinds that happened in the last hour)

                     

                    I tag traps with the alert number from the trap varbinds, and color them for the trigger and clear message. the alert manager then matches the alert-set and alert-clear traps and shows the current open alerts on the PBXes)

                     

                    I agree this is less than optimal, but it does work, is out of the box functionality, and works across upgrades (at least from 10.x through to 10.5)

                      • Re: Alert / Event from Syslog or Trap
                        wanine39

                        not being a DB expert, could the later example be made "pretty" like the first so that i could better understand the SQL logic

                         

                        also, does the later one allow parameters to be passed?

                          • Re: Alert / Event from Syslog or Trap
                            RichardLetts

                            this is one area where there are some limitations -- the SQL string has to be all on one line -- you can build it in the message writer, but I normally write it into the database editor.

                             

                            you can reformat the SQL statements at Instant SQL Formatter

                             

                            SELECT TOP 1 tv1.oidvalue + ' - ' + tv2.oidvalue + ' - '

                                         + tv3.oidvalue + ' - ' + Substring(t1.tag, 1, 20)

                            FROM   traps T1

                                   LEFT OUTER JOIN traps t2

                                                ON t1.nodeid = t2.nodeid

                                                   AND t2.colorcode = 65280

                                                   AND Substring(t1.tag, 1, 20) = Substring(t2.tag, 1, 20)

                                   INNER JOIN trapvarbinds tv1

                                           ON t1.trapid = tv1.trapid

                                              AND tv1.oidname = 'g3alarmsAlarmType'

                                   INNER JOIN trapvarbinds tv2

                                           ON t1.trapid = tv2.trapid

                                              AND tv2.oidname = 'g3alarmsMaintName'

                                   INNER JOIN trapvarbinds tv3

                                           ON t1.trapid = tv3.trapid

                                              AND tv3.oidname = 'g3alarmsPort'

                            WHERE  t1.nodeid = ${NodeID}

                                   AND t1.colorcode = 65535

                                   AND t1.datetime >= Dateadd(hour, -4, Getdate())

                                   AND t2.trapid IS NULL


                            notice the ${NodeID}  in the middle there -- that is the nodeid of the node generating this alert.

                            The outer join on T2  and then the check to see there was no match is a trick to eliminate the 'cleared' alarms (they are colored green by a rule in the trap server).

                            Note: I think it's good style to code constatnts in the join where you want to subset the data, it makes the SQL easier to read; it's also REALLY important for the logic in the outer join.

                      • Re: Alert / Event from Syslog or Trap
                        byrona

                        RichardLetts I wish I had though of this myself, flagging the traps/logs and then using a custom SQL query alert is a fantastic way of using the advanced alert engine to detect these types of failures.

                        • Re: Alert / Event from Syslog or Trap
                          SimonEvans

                          Thanks RichardLetts for providing the reset conditions for your Syslog example.

                           

                          I had been stuggling on coding a reset condition for a number of days.  When you set up a 'Custom SQL Alert' the 'standard' SolarWinds reset query condition (when 'Reset when trigger conditions are no longer true' is selected) does not work as desired because both WHERE clauses are changed to WHERE NOT clauses.

                        • Re: Alert / Event from Syslog or Trap
                          avrilspirit

                          Hi everybody,

                           

                          I realized your alert but how can I receive an alert each time I received a specific trap without reset the previous alert ?

                           

                          I test with :

                           

                          No reset condition – Trigger this alert each time the trigger condition is met  ==> it reset the older so helpdesk doesn't see the previous alert

                          No reset action - Manually remove the alert from the active alerts list ==> it makes nothing.

                           

                          I would like an alert for each syslog receive for a node.

                           

                          Thanks for your help