filtering for large numbers of devices

Tony_Grimwood

I've installed Kiwi Syslog on a server that is connected to a network of a hundred or so satellite receivers in a nationwide commercial radio broadcast network. These receivers are monitored by syslog which filters for certain events which are inherently service-impacting, i.e. receiver reboots or audio player restarts (which cause gaps in the audio). If such an event is caught, syslog sends an email to the appropriate engineering staff. This works just fine and is proving to be one of those "I don't know how we managed without it" things. There is a problem lurking here though: What happens when, for instance, sun transit occurs and all of the receivers cut over from the (temporarily silent) satellite to terrestrial backup and a hundred receivers fire off "broken audio" log messages ... A hundred emails get sent is what happens. So I need to a filter setup that will detect that a large number of these messages have arrived in a short time and send just one email. But it should also be clever enough to send an email for every such message it catches under normal conditions when the messages are few and far between. How would I do that?

 

BTW - "Sun transit" happens twice per year and lasts up to an hour per day for about two weeks each time.

 

Tony

RoyalEF

Tony, I've never found a true solution for the situation you are looking at.  It's actually quite complex, because it suggests that messages be held and counted. If more don't arrive, send the individual one; if more come, send a single, or digest of all messages accumulated.   I've sure if this was implemented it would have limits of how many. 

 

The real problem for us is ...

            do we want instant alerts -- because this is a situation that requires immediate attention; Or

            can you wait 5 minutes? 10 minutes? 60 minutes? and get a digest of all.

You likely need both (we wanted both, but settled for digests a few times a day).

 

I'm monkeyed with different permutations of a solution, but I haven't worked out the CMD-line email yet. Here's how it works:

- You write these broken audio messages to one file "BrokenAudioSyslog.txt"

- You create a SCHEDULE that runs every minutes/five minutes--whatever you can tolerate in delay about being alerted

      - it looks to see if source file exists, moves it to a history folder of these types of messages with needed time stamps in filename

      - under ARCHIVE NOTIFICATIONs you can have an email sent, but it won't attach the file or messages

              - you'll get the same generic-y email and have to go to syslog and lookup the messages yourself

      - or...under ARCHIVE OPTIONS, you can run a CMD line program

              - if you find a cmd line SMTP mailer, you could potentially send the log file as an attachment to your team

                 This is what I've hoped to find, but been unable to get a cmd-line SMTP that handled the attachments to work

                 If you try an EMAIL prog, there is likely an issue with filename you pass. You may have to COPY the file to a history folder, and leave the original to get emailed. You can delete it in a batch file that calls the EMAIL prog.

We use this method to handle temperature alerts from Cisco switches/routers.  These can come in floods.  We also use it for stack-errors on Cisco 3750 stacks.  Similarly, you get hundreds of repeating stack errors, not just one.

 

 

We get an EMAIL with subject "Kiwi Syslog Alert Report"  (note this server runs Kiwi 7.2.35)

///                 Archive Status Report                ///
------------------------------------------------------------
Date and Time:      Wed, 29 Sep 2010 14:00:02

Schedule name:      TEMP alert 14:00
Source Folder:      \Syslog\Syslogd\
Destination Folder: \Syslog\Syslogd\TEMP alarms\

+-------------------------------+------------+------+------+
| File name:                    |  File size | Move | Zip  |
+-------------------------------+------------+------+------+
| TEMP alarms-2010-09-29.txt    |    6.42 KB |  OK  | N/A  |
+-------------------------------+------------+------+------+

End of report.

 

Hope that gives you something you can put to good use.