9 Replies Latest reply: Nov 17, 2013 7:30 PM by SimonEvans RSS

Alert / Event from Syslog or Trap

poindexter

Hello thwack community,

 

Having read what I could find about the subject, I am in despair to find a way to produce visible, interpreted content in the Alert or NetPerfMon Event log in response to an SNMP Trap or Syslog entry.

 

The actions afforded in the Trap/Syslog Rule Alert Actions are divorced from the actions available in the Advanced Alert manager.  I am unable to set an alert in the way I can from a polled element.  I would like this capacity.

 

The options seem to be:

1) Create a rapid Universal Poller for the values that the traps are reporting on

     This does nothing for Syslog

2) Launch an external program / script that can insert Alerts and / or Event entries

 

Option 1 is pretty silly, generating a lot of traffic and introducing delay in response.

 

Option 2 seems to be the way to do it, but the Orion SDK documentation does not make it clear what is an is not accessible through a CRUD/SWIS operation.  Here are some potential strategies:

 

2a) Do a managed (via some SWIS API call) triggering of an alert.

2b) Do a direct (via SWQL) insertion of an Alert or Event.  Has this been done?  Is it possible?

2c) Set a custom property on a node indicating the current error

     This would require a universal poller for the Advanced Alert Manager to pick up... with a high polling rate. =\

 

Any help would be appreciated.

 

P.S.: Does anyone else find it a little ridiculous that SNMP traps and Syslog entries are not able to trigger alerts out of the box?

 
  • Re: Alert / Event from Syslog or Trap
    RichardLetts

    I've posted on this topic extensively, and there are several ideas out there for improving the alert engine. the Trap and syslog processors work in real-time on incoming messages, whereas the alert manager runs asynchronously based on what is in the database. You have to think hard about how this is going to work together (if it were easy SW would already have fixed it)

     

    Right now I think the best solution is to use custom SQL alerts.

     

    syslog:

     

    in the syslog engine I match messages and set the messagetype; this allows me to flag similar syslog messages with the same message type. Unfortunately you can't export these or I'd include an example.

    Match: *UPS: An inverter fault exists. 0x0165*,*UPS: In bypass in response to an internal hardware fault. 0x011B*

    Modify the syslog, set the message type = 'UPS_ERROR_INTERNAL',

     

    the advanced alert rule then becomes a custom SQL alert on a node with the following condition

    where nodeid in (
    SELECT nodeid
    FROM   syslog
    WHERE  messagetype ='UPS_ERROR_INTERNAL'
           AND datetime > Dateadd(day, -1, Getdate())
    )
    
    

     

    and the reset condition is:

    where nodeid NOT in (
    SELECT nodeid
    FROM   syslog
    WHERE  messagetype ='UPS_ERROR_INTERNAL'
           AND datetime > Dateadd(day, -1, Getdate())
    )
    
    

     

    For traps I do a similar thing:

    match trap conditions, set a TAG in the trap rule, and then use a custom SQL alert like the following:

     

    WHERE  nodeid IN (SELECT nodeid
                      FROM   traps
                      WHERE  datetime>Dateadd(hour,-1, Sysdatetime())
                      and tag='OSPF_IF_CONFIG_ERROR'
                     )
    
    

     

    /RjL

    • Re: Alert / Event from Syslog or Trap
      poindexter

      Yes, I've read your posts in regards to this matter, and you're advocating option 2c here.

       

      It's not ideal.  There is data within the trap that I would need to pass to the alert, and it doesn't seem like the Tag action has the ability to extract information from the trap.

       

      Do you have any experience using the "SDK"?

      • Re: Alert / Event from Syslog or Trap
        RichardLetts

        no, I'm not advocating option 2c -- I'm advocating the option you don't list. The syslog and trap receivers process the values as they come in. You don't need a universal device poller at all, you can sumply do a custom SQL alert in the alert manager.

         

        You can tag the trap with anything that it has extracted from the trap, e.g. ${vbData5} for the 5th varbind in the trap.

         

        I use the SDK extensively; I don't see that it will help you here.

         

        Why not try my suggestion and share where you get stuck.

         

        /RjL

        • Re: Alert / Event from Syslog or Trap
          poindexter

          I see, it doesn't require a poller to do SQL alerts.  The variable extraction is exactly what I need, thanks!

           

          Edit:

           

          Although this is the correct answer, it is more complex (and likely resource intensive) than simply using a fast universal poller on the relevant SNMP properties.

      • Re: Alert / Event from Syslog or Trap
        RichardLetts

        oh, if you need specific things from a trap then you can do something like this in the alert advanced alert actions

        write a message to the event log with the following text :

        PBX alert on node: ${Caption} ${SQL:SELECT top 1 tv1.oidvalue +' - '+ tv2.oidvalue + ' - ' + tv3.oidvalue + ' - ' + Substring(t1.tag, 1, 20) FROM traps T1 left outer join traps t2 on t1.nodeid=t2.nodeid and t2.colorcode=65280 and Substring(t1.tag, 1, 20)=Substring(t2.tag, 1, 20) inner join trapvarbinds tv1 on t1.trapid=tv1.trapid and tv1.oidName='g3alarmsAlarmType' inner join trapvarbinds tv2 on t1.trapid=tv2.trapid and tv2.oidName='g3alarmsMaintName' inner join trapvarbinds tv3 on t1.trapid=tv3.trapid and tv3.oidName='g3alarmsPort' WHERE t1.nodeid = ${NodeID} and t1.Colorcode= 65535 AND t1.datetime >= Dateadd(hour, -4, Getdate()) and t2.trapid is null  }

        (which extracts the last non-cleared Avaya PBX alert type from the trapvarbinds that happened in the last hour)

         

        I tag traps with the alert number from the trap varbinds, and color them for the trigger and clear message. the alert manager then matches the alert-set and alert-clear traps and shows the current open alerts on the PBXes)

         

        I agree this is less than optimal, but it does work, is out of the box functionality, and works across upgrades (at least from 10.x through to 10.5)

        • Re: Alert / Event from Syslog or Trap
          wanine39

          not being a DB expert, could the later example be made "pretty" like the first so that i could better understand the SQL logic

           

          also, does the later one allow parameters to be passed?

          • Re: Alert / Event from Syslog or Trap
            RichardLetts

            this is one area where there are some limitations -- the SQL string has to be all on one line -- you can build it in the message writer, but I normally write it into the database editor.

             

            you can reformat the SQL statements at Instant SQL Formatter

             

            SELECT TOP 1 tv1.oidvalue + ' - ' + tv2.oidvalue + ' - '

                         + tv3.oidvalue + ' - ' + Substring(t1.tag, 1, 20)

            FROM   traps T1

                   LEFT OUTER JOIN traps t2

                                ON t1.nodeid = t2.nodeid

                                   AND t2.colorcode = 65280

                                   AND Substring(t1.tag, 1, 20) = Substring(t2.tag, 1, 20)

                   INNER JOIN trapvarbinds tv1

                           ON t1.trapid = tv1.trapid

                              AND tv1.oidname = 'g3alarmsAlarmType'

                   INNER JOIN trapvarbinds tv2

                           ON t1.trapid = tv2.trapid

                              AND tv2.oidname = 'g3alarmsMaintName'

                   INNER JOIN trapvarbinds tv3

                           ON t1.trapid = tv3.trapid

                              AND tv3.oidname = 'g3alarmsPort'

            WHERE  t1.nodeid = ${NodeID}

                   AND t1.colorcode = 65535

                   AND t1.datetime >= Dateadd(hour, -4, Getdate())

                   AND t2.trapid IS NULL


            notice the ${NodeID}  in the middle there -- that is the nodeid of the node generating this alert.

            The outer join on T2  and then the check to see there was no match is a trick to eliminate the 'cleared' alarms (they are colored green by a rule in the trap server).

            Note: I think it's good style to code constatnts in the join where you want to subset the data, it makes the SQL easier to read; it's also REALLY important for the logic in the outer join.

    • Re: Alert / Event from Syslog or Trap
      byrona

      RichardLetts I wish I had though of this myself, flagging the traps/logs and then using a custom SQL query alert is a fantastic way of using the advanced alert engine to detect these types of failures.

    • Re: Alert / Event from Syslog or Trap
      SimonEvans

      Thanks RichardLetts for providing the reset conditions for your Syslog example.

       

      I had been stuggling on coding a reset condition for a number of days.  When you set up a 'Custom SQL Alert' the 'standard' SolarWinds reset query condition (when 'Reset when trigger conditions are no longer true' is selected) does not work as desired because both WHERE clauses are changed to WHERE NOT clauses.