9 Replies Latest reply: Feb 2, 2009 7:49 AM by rrichardson RSS

Limiting Log Retention

Hello.

 

I've installed the free version of Kiwi Syslog (I'm a long-time user of CatTools), and am unable to find a setup preference which tells Kiwi how long to retain syslog messages.  I don't have unlimited drive space, and only want to keep certain messages for a limited period.

 

More specifically, need to keep the NAT translation messages from my firewall, so I can track down inappropriate use by students.  These messages come at a rate of over 20,000/hr.  I only want to keep them for a week.

 

Thanks

 
  • Re: Limiting Log Retention

    Kiwi doesn't have log retention rules or limits.

    I'm using 7.2.35 (registered) and we manually remove firewall logs every month.  We rely on the disk space alert to let us know when it is time.

    Our firewall Syslog server takes in 6 to 7 million messages per hour, and we log log over 25 Gb per day on a busy day. We retain 7 weeks to 4 months of logs, depending upon time of year.

    One trick is to use Compressed folders (Windows 2003 server) for the syslogd log directory.  We find the low CPU impact of compressing the folders gives back a lot in disk throughput and disk space since it reduces disk writes/reads by about 75%.  Those 25 Gbs only takes up 7.5Gb on disk, which is 75% less disk activity and space.

    • Re: Limiting Log Retention
      chris.lapoint

      I'm curious to know if you ever tried to make this work using the scheduled archive feature and if so, what were the limitations that made this not possible.  For example, is it because the size of the firewall logs are not predicatable enough to implement a scheduled archive/deletion?

      • Re: Limiting Log Retention
        chris.lapoint

        I just realized you're on a really old version!  7.2.35   Log file rotation was added in 8.30 (http://www.kiwisyslog.com/index.php?option=com_changelog&Itemid=119&ver=8.3.0).  We also added a new scheduler that replaces the old archive functionality in that version.  See screen shots below.   When you get a chance, please download the latest version and let us if that meets your needs.

        • Re: Limiting Log Retention
          chris.lapoint

          Screenshot of log file rotation settings

        • Re: Limiting Log Retention

          Thanks, Chris I'll take a look at the feature set.   From the screen shot it doesn't look like it would fit our needs.  From a security and accountability perspective we keep as much as we can throughout the year.  So sometimes that is four months, sometimes seven weeks.   We delete only when space forces us to.  The options I see at first glance focus on hard set ## days old or log files of ### size. 

          Because none of the new Kiwi features had been must-haves in my past reviews management at this customer site has decided to stay put.  There is risk of new problems or reduced throughput from new features increasing the program's overhead.   And that last part is critical for us.  If 8.3 has a higher CPU impact than 7.2 then an upgrade will destroy us.

          It is also difficult to test loads becuase Kiwi's syslog forwarding is very intensive.  When we attempted to forward syslog to a second new server to do load testing Kiwi maxxed out the CPU on the existing installation and we began loosing messages.  And that is with a configured message overflow buffer of 200,000 messages!

          So changes and testing is a risky move and this is the most risk-averse, techno-phobic large scale I.T. organization I have ever dealt with. They are frightened like children of making changes. It's quite bizarre.

      • Re: Limiting Log Retention

        "...to implement a scheduled archive/deletion?"

         

        Is there a deletion feature in the schedule?  I couldn't find it.

        • Re: Limiting Log Retention

          The first two choices that come to mind to try is either implement a Log rotation in the logging command, or

          Archive aged out files to a "TO BE DELETED" folder, and after the Archive RUN PROGRAM of a batch file that deletes "C:\TO BE DELETED\*.*"

           

          I would opt to use the later, since the LOG ROTATION is a very simplistic feature that nullifies such features as insert DATE-TIME variables into the title of the Log file.  We do this with every log file, as it is impractical (except in little shops) to use something like a SyslogCatchAll.txt file.

          Once you start dating the files, then I would assume the Log Rotation would Rotate Max_N number of uniquely named logs:

          Using AutoInsert of a date (i.e."2009-01-20_ExternalFW") would rotate through Max_N number of log files per day.  I'm not sure what it appends to differentiate. So each day in this AutoInsert example, I assume, would have up to N logs.  

          I see limited use for Log Rotations, because (from my assumptions) the AutoInsert already is much more powerful than that.  I certainly see that people may want to stop from giant log files.  We do have some log files that are over 100 Mbs, and we have wondered about splitting them by hour.  Others are already split by hour and have exceeded 60 mb, and we waffle on whether to split by the minute.  Log rotation would give us an inbetween there.  We could rotate the hourly stamped logs by 10 megabyte chunks.  I could see implementing that.

          But that doesn't help with eliminating aged logs.  The archive suggestion would.  We have played around with writing VBS scripts that when disk went below a certain margin, would find the oldest dated log files and delete them.

          • Re: Limiting Log Retention
            Kuz

            Sounds like you guys are forgetting the Scheduled Clean-up task.

            It's purpose is to delete log files that:
            (a) Match a given file mask.
            and (b) Are of a certain size, or age.

            Check out the Scheduled Clean-up task, by creating a new Schedule and changing the task type to "Clean-up".  The Source page contains all the config required to establish some log-retention rules.

            More info on the Scheduled Clean-up task here:  http://www.kiwisyslog.com/help/syslog/clean_uptask.htm

            Mike Kuzman, Dev. Lead
            Kiwi Syslog Server
            Solarwinds
            New Zealand