cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 13

syslog message type summary, and alerting on rarity

I would like to see Kiwi (and Orion syslog for that matter) be able to include a daily message type summary along with the statistics that we can already get. Terry Slattery has a post on this feature on his blog:

http://connection.netcordia.com/blogs/terrys_blog/archive/2009/10/28/syslog-summary-script.aspx

This would be incredibly useful to find syslog messages that occur rarely.

Even better would be the ability to fire an alert on a syslog message type that has never been seen before, or to search by "rarity" (I believe Splunk has this feature).

0 Kudos
4 Replies
Level 12

Hi jswan,

You could combine a couple of scripts that we have:

The first script needs to be added to the default rule, and records (in a scripting dictionary) a count of messages recieved per host (IP address).  See [Script_HostCount.txt] attached. (VBScript, requires full read/write permissions)

The second script generates an email of the dictionary contents and then clears the dictionary.  This can be run once per day (or however often you like).  To get the script to run over a given interval (say, every 24 hrs), you will need to:
1) Enable a 24-hour Keep-alive message.  (Setup > Inputs > Keep-alive)
Frequency = 86400 seconds (24 hrs)
2) Create a new rule, with one filter: "Input Source", with just "Keep-alive" selected.  (This filter will "catch" the 24-hour keep-alive event, and allow you to run the second script).
3) Add the Run-Script action to the rule.  See [Script_HostCountMailandReset.txt] attached. (VBScript, requires full read/write permissions)

To sumamrize:

Default Rule
+filters
+actions
 -Display
 -Log to file
 -RunScript [Script_HostCount.txt]

New Rule
+filters
 -InputSource = "Keep-alive"
+actions
 -RunScript [Script_HostCountMailandReset.txt]


Let me know if you have any problems getting it to work.  You will need to modify Script_HostCountMailandReset.txt to include your mail server address and e-mail address, etc.

0 Kudos

I finally finished migrating my syslog server and now I have a chance to look at this closely. As far as I can tell, this script only counts the number of syslog messages per host, not the number of each message type.

I'm looking for something that shows the count of each message type. Specifically, I'd define the message type as the part from the % to the : in a Cisco IOS syslog message. For example, I'd like to see counts that look like:

%C6KERRDETECT-2-FIFOCRITLEVEL:1

%LINEPROTO-5-UPDOWN: 300

%CDP-4-DUPLEX_MISMATCH:500

Ideally, these would be sorted in order of increasing count, so I can see what rare messages occurred.

Even better would be to keep an ongoing tally of message types received, and create a separate warning with any new message types that haven't been seen before:

Warning, message type:

%C6KERRDETECT-2-FIFOCRITLEVEL:1

occurred on device X and has never been seen before. Complete message text:

--complete message text here--

 

etc.

0 Kudos

Hi jswan,

Adding support for interesting message types to the script isn't difficult.

Try making the following changes to Script_HostCount.txt.   (This will keep counts for the three message types you are interested in.  To include other message types, add them to the ... messageType(N) = "xxxxx" ... section of the script).

Function Main()

 

' This function keeps a record of the number of syslog messages received from different message types

'

' 'Stats' dictionary holds count for each message type

 

 

'** Edit these Message Types **

 

Dim messageType(2)

  messageType(0) = "%C6KERRDETECT-2-FIFOCRITLEVEL:1"

  messageType(1) = "%LINEPROTO-5-UPDOWN:300"

  messageType(2) = "%CDP-4-DUPLEX_MISMATCH:500"

 

'** Edit these Message Types **

 

Dim messageText

messageText = Fields.VarCleanMessageText

Dim thisMessageType

 

thisMessageType = ""

For i = 0 to UBound(messageType)

If Instr(messageText, messageType(i))>0 Then

thisMessageType = messageType(i)

End If

Next

 

If thisMessageType<>"" Then

With Dictionaries

if .Exists("Stats") then

If .ItemExists("Stats", thisMessageType) Then

cnt = .GetItem("Stats", thisMessageType)

.StoreItem "Stats", thisMessageType, (cnt + 1)

Else

.StoreItem "Stats", thisMessageType, 1

End If

Else

.StoreItem "Stats", thisMessageType, 1

End If

End With

End If

 

' Return 'OK' (success)

Main="OK"

 

End Function

0 Kudos

The problem with that solution is that I don't know in advance what messages I want to count. I have a *nix scripting background rather than a Windows one, so I ended up solving this by using a shell script completely outside Kiwi. I installed Cygwin on the Kiwi server and built two shell scripts:

#!/bin/bash
/usr/bin/grep % Logs/SyslogCatchAll.txt | /usr/bin/awk '{print $4,$10}' | /usr/bin/sort | /usr/bin/uniq -c | /usr/bin/sort -n | /usr/bin/email -s "Syslog Summary by Source IP" jswan@foo.com -f noc@foo.com -n "Kiwi Syslog Server" --smtp-server 1.2.3.4


#!/bin/bash
/usr/bin/grep % Logs/SyslogCatchAll.txt | /usr/bin/awk '{print $10}' | /usr/bin/sort | /usr/bin/uniq -c | /usr/bin/sort -rn | /usr/bin/email -s "Syslog Summary by Type" jswan@foo.com -f noc@foo.com -n "Kiwi Syslog Server" --smtp-server 1.2.3.4

Then I set these up to run as a scheduled task at 23:59:30 every day.

These let me see what message types are produced daily, overall, and which message types are most frequently produced by which devices.

This only works for Cisco syslog messages (that's why I grep for %), which are formatted consistently enough that awk can grab the IP address and message type fields reliably.

0 Kudos