12 Replies Latest reply on May 31, 2013 10:47 AM by michael2907

    Garbage In, Garbage Out

    Mrs. Y.

      One log correlation vendor has what I colorfully refer to as a “heroin dealer licensing model.” In other words, the first taste is for free, but then get ready to hand over your checkbook. It’s a good product, but it’s based upon the idea that you can reliably predict the daily size of your log files with great accuracy. Please raise your hand if you’ve ever had one of those late-night messages on your phone alerting you that a volume ran out of space, even though you carefully rotate your logs on a weekly or even daily basis. Even in a well-managed organization, there will be times where you get surprised because of an unusual system event, so you’re in a constant state of fear about storage. I, for one, am tired of garbage data that clogs up my garbage monitoring and alerting systems with meaningless messages. Sartre was wrong, Hell isn’t other people, it’s your log and event correlation system.

        • Re: Garbage In, Garbage Out
          byrona

          I, for one, am tired of garbage data that clogs up my garbage monitoring and alerting systems with meaningless messages.

          If it's your garbage monitoring wouldn't you expect garbage data? 

           

          I think when we begin to talk about these things we need to classify or define what is actual garbage (unimportant and unneeded data) versus "garbage data" that can have valuable information in it.  I think if we can classify the difference between the two it shouldn't be too difficult to have the system do the same and throw out (or never collect) the stuff we don't care about and only maintain the stuff that we do care about.

          • Re: Garbage In, Garbage Out
            Chet Camlin

            It is a little difficult to translate your issue but since this is the General Security and Compliance area, I’m going to assume you are talking about a security or compliance system designed to collect and store logs from many devices and the logging level is turned up fairly high. In this case you may not be able/allowed to filter the data collected.


            Assuming you cannot adjust the level of logging or the system/devices being logged, I’d turn my attention to the physical log drive.  If you are able to fill up your log drive in one day, it’s time to increase your log drive size. 

            If you cannot adjust your log drive size, I’d setup alerting in NPM to give me a count down of the drive space used.  Set up 5 alerts. Set one for “Volume percent used is greater or equal 50%”, one for “Volume percent used is greater or equal 60%”, one for 70%, 80% and 90%. Have the alert email, SMS, or some action that you or your staff will notice.  This will give you and indication of how fast the log drive is filling up.  You stated that you have to manually empty the log drive.  I would create a batch file that would move the files to another drive with plenty of space automatically. Set this batch file to execute as a trigger action for 90% alert.

             

            I looked at setting up a nested condition to evaluate volume percentage used under trigger conditions. I believe it is possible by using the ANY condition and specifying a nested condition group with 2 conditions. One evaluating say greater than 50% and the next less than 59%. I’ll have to sit down and test this solution.   

             

            The NPM alerting system is very useful. Hope this helps. If not, ignore all before this sentence.

            • Re: Garbage In, Garbage Out
              Webbster

              To reduce the number of logs being generated we adjusted the logging level to just report on items that were relevant to our needs (i.e. no informational items).

              • Re: Garbage In, Garbage Out
                Aaron Denning

                we do the same as webbster we adjusted to what was needed.

                • Re: Garbage In, Garbage Out
                  xbod

                  How about having 2 log collectors or 1 collector and 2 storage areas.  One for the logs you want to save and one for the logs you have to save.  The stuff we want is what it takes to do our jobs.  Logs that help us troubleshoot and point to problems.  The stuff we have to save is what someone says is necessary for auditing or CYA moments.

                  I imagine the 2nd category requires more storage based on the longer retention time while the 1st category of logs may not need to be retained for as long and may not need an archiving solution.

                  • Re: Garbage In, Garbage Out
                    gallifreyan

                    I would guess you refer to the product that rhymes with the sound your head makes when you realize how much that volume of logs will cost you... kerplunk?

                     

                    One common use case for "Big Data(tm)" is to gather the data before you know what it is or how/why/if you want/need it. But that should eventually lead to figuring out how/why/if, and aging out the data. I'm not sure how Kerplunk handles this; it's been years since I dealt with them. But another idea is to consider a platform like Logstash for your greasy grimy gopher logs, and put your high touch data into Kerplunk.

                    • Re: Garbage In, Garbage Out
                      dafludd

                      I agree 100% I am sick of getting garbage alerts and things waking me up in all hours of the night.

                        • Re: Garbage In, Garbage Out
                          mdriskell

                          I've been in charge of various event management systems for the last 7 years and it's a constant struggle to make sure that alerts contain not only the right amount of data but relevant data.  At some point even if valid if a team is getting 3000 alerts in a month they can't be reading each one and perhaps we need to circle back.

                        • Re: Garbage In, Garbage Out
                          dougeria

                          Is it just a matter of logging the right events?

                          • Re: Garbage In, Garbage Out
                            zackm

                            Bingo.

                             

                            The 'right' event is like the 'right' flavor of ice cream.

                             

                            When I worked as a contractor with the federal government, we had policies in place requiring that we keep ALL logs captured in SIEM for some absolutely ridiculous retention periods. And we aren't talking about the cool "super secret squirrel spy stuff" as seen on TV. No, we're talking about "not even juicy enough to be classified" data.

                             

                            'Right' is in the eye of the InfoSec director unfortunately. But, I do believe it was Sartre who also said "A lost battle is a battle one thinks one has lost"

                             

                             

                              • Re: Garbage In, Garbage Out
                                michael2907

                                I've had somewhat similar experiences. I've done some beta testing for file auditing systems, but that's one thing I don't think I'll do again without a few specific certainties. I was in a situation where a few bugs happened to send the log system out of whack, and the size of those log files increased exponentially. Somehow it managed to fill up an entire 1 TB of space with plaintext logs, and here I am in the middle of the night getting an email alert every few seconds when a lot of our backups tried to hit that fileserver. Talk about frustration.