4 Replies Latest reply on Jan 28, 2010 10:30 AM by jswan

    syslog message type summary, and alerting on rarity

    jswan

      I would like to see Kiwi (and Orion syslog for that matter) be able to include a daily message type summary along with the statistics that we can already get. Terry Slattery has a post on this feature on his blog:

      http://connection.netcordia.com/blogs/terrys_blog/archive/2009/10/28/syslog-summary-script.aspx

      This would be incredibly useful to find syslog messages that occur rarely.

      Even better would be the ability to fire an alert on a syslog message type that has never been seen before, or to search by "rarity" (I believe Splunk has this feature).

        • Re: syslog message type summary, and alerting on rarity
          Kuz

          Hi jswan,

          You could combine a couple of scripts that we have:

          The first script needs to be added to the default rule, and records (in a scripting dictionary) a count of messages recieved per host (IP address).  See [Script_HostCount.txt] attached. (VBScript, requires full read/write permissions)

          The second script generates an email of the dictionary contents and then clears the dictionary.  This can be run once per day (or however often you like).  To get the script to run over a given interval (say, every 24 hrs), you will need to:
          1) Enable a 24-hour Keep-alive message.  (Setup > Inputs > Keep-alive)
          Frequency = 86400 seconds (24 hrs)
          2) Create a new rule, with one filter: "Input Source", with just "Keep-alive" selected.  (This filter will "catch" the 24-hour keep-alive event, and allow you to run the second script).
          3) Add the Run-Script action to the rule.  See [Script_HostCountMailandReset.txt] attached. (VBScript, requires full read/write permissions)

          To sumamrize:

          Default Rule
          +filters
          +actions
           -Display
           -Log to file
           -RunScript [Script_HostCount.txt]

          New Rule
          +filters
           -InputSource = "Keep-alive"
          +actions
           -RunScript [Script_HostCountMailandReset.txt]


          Let me know if you have any problems getting it to work.  You will need to modify Script_HostCountMailandReset.txt to include your mail server address and e-mail address, etc.

            • Re: syslog message type summary, and alerting on rarity
              jswan

              I finally finished migrating my syslog server and now I have a chance to look at this closely. As far as I can tell, this script only counts the number of syslog messages per host, not the number of each message type.

              I'm looking for something that shows the count of each message type. Specifically, I'd define the message type as the part from the % to the : in a Cisco IOS syslog message. For example, I'd like to see counts that look like:

              %C6KERRDETECT-2-FIFOCRITLEVEL:1

              %LINEPROTO-5-UPDOWN: 300

              %CDP-4-DUPLEX_MISMATCH:500

              Ideally, these would be sorted in order of increasing count, so I can see what rare messages occurred.

              Even better would be to keep an ongoing tally of message types received, and create a separate warning with any new message types that haven't been seen before:

              Warning, message type:

              %C6KERRDETECT-2-FIFOCRITLEVEL:1

              occurred on device X and has never been seen before. Complete message text:

              --complete message text here--

               

              etc.

                • Re: syslog message type summary, and alerting on rarity
                  Kuz

                  Hi jswan,

                  Adding support for interesting message types to the script isn't difficult.

                  Try making the following changes to Script_HostCount.txt.   (This will keep counts for the three message types you are interested in.  To include other message types, add them to the ... messageType(N) = "xxxxx" ... section of the script).

                  Function Main()

                   

                       ' This function keeps a record of the number of syslog messages received from different message types

                       '

                       ' 'Stats' dictionary holds count for each message type

                   

                   

                       '** Edit these Message Types **

                   

                       Dim messageType(2)

                        messageType(0) = "%C6KERRDETECT-2-FIFOCRITLEVEL:1"

                        messageType(1) = "%LINEPROTO-5-UPDOWN:300"

                        messageType(2) = "%CDP-4-DUPLEX_MISMATCH:500"

                   

                       '** Edit these Message Types **

                   

                       Dim messageText

                       messageText = Fields.VarCleanMessageText

                             

                       Dim thisMessageType     

                   

                       thisMessageType = ""

                       For i = 0 to UBound(messageType)

                            If Instr(messageText, messageType(i))>0 Then

                                 thisMessageType = messageType(i)

                            End If

                       Next

                   

                       If thisMessageType<>"" Then

                            With Dictionaries

                                 if .Exists("Stats") then

                                      If .ItemExists("Stats", thisMessageType) Then

                                           cnt = .GetItem("Stats", thisMessageType)

                                           .StoreItem "Stats", thisMessageType, (cnt + 1)

                                      Else

                                           .StoreItem "Stats", thisMessageType, 1

                                      End If

                                 Else

                                      .StoreItem "Stats", thisMessageType, 1

                                 End If

                            End With

                       End If

                   

                       ' Return 'OK' (success)

                       Main="OK"

                   

                  End Function

                    • Re: syslog message type summary, and alerting on rarity
                      jswan

                      The problem with that solution is that I don't know in advance what messages I want to count. I have a *nix scripting background rather than a Windows one, so I ended up solving this by using a shell script completely outside Kiwi. I installed Cygwin on the Kiwi server and built two shell scripts:

                      #!/bin/bash
                      /usr/bin/grep % Logs/SyslogCatchAll.txt | /usr/bin/awk '{print $4,$10}' | /usr/bin/sort | /usr/bin/uniq -c | /usr/bin/sort -n | /usr/bin/email -s "Syslog Summary by Source IP" jswan@foo.com -f noc@foo.com -n "Kiwi Syslog Server" --smtp-server 1.2.3.4


                      #!/bin/bash
                      /usr/bin/grep % Logs/SyslogCatchAll.txt | /usr/bin/awk '{print $10}' | /usr/bin/sort | /usr/bin/uniq -c | /usr/bin/sort -rn | /usr/bin/email -s "Syslog Summary by Type" jswan@foo.com -f noc@foo.com -n "Kiwi Syslog Server" --smtp-server 1.2.3.4

                      Then I set these up to run as a scheduled task at 23:59:30 every day.

                      These let me see what message types are produced daily, overall, and which message types are most frequently produced by which devices.

                      This only works for Cisco syslog messages (that's why I grep for %), which are formatted consistently enough that awk can grab the IP address and message type fields reliably.