5 Replies Latest reply: Jun 8, 2011 10:48 AM by abailey25 RSS

    Syslog Retention and Archiving

      Recently I have been tasked to improve our log monitoring and retention program within our organization.

      During my research on this program, I discovered that our organization has a requirement to maintain log data using the following as a guide:

      - 30 days online
      - 1 year offline (archiving)

      The 30 day retention online is a no brainer.  The Syslog monitor tool was simple enough to configure although we are considering upgrading our server drive space to provide more wiggle room for the database environment.

      The offline requirement is becoming a bit of a question around here.
      Are there any practical best practice approaches to archiving that people are using out there?

      Ideas we brainstormed and are looking at:

      • dual log shipping; ship logs to NPM for 30day monitoring and ship logs to another server for dedicated raw archiving to a biweekly tape to be held for a year.
      • back up the NPM database weekly for retention of data for one year (many solutions to accomplish this as long as a year of data is held)

      We are still learning here, so any information, ideas, or advice is much appreciated.

      Cheers!

        • Re: Syslog Retention and Archiving
          bshopp

          Just an idea to throw out other customers have done.  Not sure how you are setup so this may not apply to you.

          Deploy our Kiwi Syslog Servers at regional sites and have those devices at that site log to that server.  Kiwi can archive off on a pre-set basis automatically and you can set it up to automatically forward all or specific messages up to Orion for online viewing

            • Re: Syslog Retention and Archiving
              savell

              This is the approach we use (but mainly for security reasons rather than archiving).

              It will work much better once Orion can strip out the orgin_address from a Kiwi syslog forwarded message.

              This was listed as one of the items under development - hopefully it made the cut for the next release.

              Dave.

              • Re: Syslog Retention and Archiving

                We recently had a discussion regarding your suggestion.

                Our environment is relatively small.  I estimate that only 100-150 devices are in play in regards to log archiving requirements.

                The question that came up:

                Is there was a way to archive what is shipped directly to the Orion server without using a "middle-man" implementation like you suggest?

                Initial group thought is that we define our 1 year offline archive as the database backups of our Orion server and leave it as that.  The database backups could be archived offline for a year.

                My contention with this approach is twofold.

                1) The amount of resources and effort to manage this backup library may become too much to handle effectively.  This is yet to be determined or estimated.

                2) The amount of work it would take to go through this database backup archive if forensic analysis is required in the future.  A raw syslog archive would be much easier to analyze quickly if we do find ourselves in a time sensitive investigative mode.

                The implementation of a Kiwi Syslog server to split out the log data in two directions may not be an option due to resouce limitations on our part.  We would prefer that the products we already have would be able to provide archive capability in some fashion.

                More to come on this...  Ideas we are still playing with:

                1) Dual homing the syslog shipping on network devices. We are not certain if this is a viable solution on all devices.  On Windows server systems, we would continue to use the syslog generation tool provided with Orion and also use scripting on each server to copy the raw Windows logs of each server logs on our Windows systems to an archive that is loaded to offline archives on a established cycle.

                BTW: You may already know this, but just to give a background to those who do not... 
                This archiving requirement is not arbitrary on our part.  Our organization falls under DISA regulation and policy and the 30 day online, 1 year offline requirement is specified as a standard quite often in several of the DISA Security Technical Implementation Guides(STIG).  STIG specifications are commonly used for C&A validation in regards to auditing within DoD organizations.
                We are meeting this requirement, but barely.  To us, the Orion tools will allow use to actually use this data more effectively.  Now if we can just find a more manageable way to deal with archiving logs, we would be golden.

                • Re: Syslog Retention and Archiving
                  abailey25

                  I have Orion NCM 5.5.2 and am trying to accomplish the same thing.  Is there a way I can archive logs with the software I already have? Can Orion NCM syslog forward logs to a kiwi syslog free edition somehow? It would be nice if Orion NCM syslog had a archive option where it could export syslog data to a file on a schedule for archiving.