21 Replies Latest reply on May 31, 2013 10:51 AM by michael2907

    Log Correlation: There Can Be Only One

    Mrs. Y.

      When I talk about log correlation with co-workers, I frequently use the analogy of "Highlander." For those of you who aren't sci-fi nerds, this was a movie (and later a TV show) in which the protagonist, an immortal named Conner Macleod, seeks absolute power and knowledge. Without going into too much detail, that means beheading other immortals in order to obtain their life-force in a process called a "quickening," in an attempt to reach the final goal of becoming the "only one."

       

       

       

      I find the similarity between this story and centralized logging very appropriate. As your log and event server ingests more data, it becomes increasingly omniscient, just like one of those immortals. You don't have to run around as much trying to pull data from various systems in order to conduct a forensic investigation or perform troubleshooting and you can even anticipate issues proactively. However, like the characters in "Highlander," bad information or a “dark quickening” can also overwhelm a centralized system. Think about all those alerts and events you start to ignore, the ever-increasing disk storage required. This occurs because someone neglects to spend the time up front to determine baselines and performance thresholds, create data retention policies and fails to tune the system as it matures. With log and event correlation, just like “Highlander,” an immortal’s work is never done and there’s always more knowledge to consume, one more nemesis to behead. After all, the movie did spawn multiple sequels and two TV series.

        • Re: Log Correlation: There Can Be Only One
          mdriskell

          First of all I should state that at one point in time There Can Be Only One was my network wifi password with some special characters thrown in for good measure of course.  Always loved that movie.

           

          As you stated in a previous post it's all about the right data.  It all comes back to design.  Like the old saying goes measure twice cut once.  By spending the time to design this system (or any other for that matter) properly you will get the most out of it.

          • Re: Log Correlation: There Can Be Only One
            Chet Camlin

            To avoid the avalanche of events that no one can decipher, I’ve always used targeted monitoring.  I’d ask myself “what are the most common problems associated with my network”. To implement targeted monitoring, you must understand why the problem occurred (root cause).  Too often a knee jerk solution is put into effect based on what failed. Example: an SMTP service is failing 4 times a week so lets just monitor the SMTP service and have a script restart it if it fails.  Wrong answer. Why did it fail?  Is this a symptom of a bigger problem?

             

            A simple well tuned monitoring system (NPM and LEM) and a well-trained, experienced admin will out perform a high cost complicated monitoring system.  When it comes to my network, I’d rather have a human tell me what happened based on empirical data supplied by the monitoring system. 

            • Re: Log Correlation: There Can Be Only One
              byrona

              Or if you prefer a different flavor of geekery "The one to rule them all". 

               

              I would certainly agree, when it comes to log and event management you really need to have just one spot, even if you are just using it for forensics and not doing anything proactive.  Having the ability to go to one place where you know all of your data is stored and having the ability to search through that data efficiently and see what was happening across all of your different devices at any point in time is absolutely necessary.

              • Re: Log Correlation: There Can Be Only One
                superfly99

                I'm definitely not a sci-fi nerd but I agree that all logs should be in the one place and also that we capture what we believe is important. There will be times when I'm not monitoring the exact thing that broke. Capturing logs will always be tuned as if we capture absolutely everything, we'll never be able to decipher it all.

                • Re: Log Correlation: There Can Be Only One
                  zackm

                  Another perspective is that our world is becoming more and more consolidated with companies being acquisitioned left and right (Hello, N-Able). Poor planning in a small environment can quickly and exponentially escalate into an out of control situation once a traditional enterprise environment is added to your loggers. The mantra should be that we always plan for the biggest we can plan for, while staying within budget of course.

                  • Re: Log Correlation: There Can Be Only One
                    dougeria

                    Centralized reporting can also cause single point of failure if not configured correctly.  Poor planning due to small budgets and limited time are always going to be an issue.

                    • Re: Log Correlation: There Can Be Only One
                      ssei

                      There are two things that I try to keep in mind with the volume of events and alerts. First, if there are too many false alarm alerts, they tend to become ignored. Secondly, producing meaningful reports from the volume of events is critical to performing good log reviews.

                        • Re: Log Correlation: There Can Be Only One
                          Kellie Mecham

                          OK, I know this phrase "producing meaningful reports from the volume of events is critical to performing good log reviews" is true, what' I'd love to know is, perfect world, what does that mean for you?  What would make something meaningful and useful to you?  And, not just you but other possible users in your environment (maybe CTO or CISO, maybe auditors,maybe just IT Director).  Any elaboration on kinds of report details and/or types of users, I'd love to know more.

                            • Re: Log Correlation: There Can Be Only One
                              jeremymayfield

                              Meaningful reports, something that might be easily customized and delivered in to the necessary hands.   Reports should lend itself easily to business and not just IT needs for the purpose of driving effective change to business stake holders.  Aid in data dumping to be easily combined with other reporting engines like Crystal reports used with or standalone for Business and Fiscal needs.  Be well automated, while being maintained on flexible schedules.  I would like to see reports that mean something to me, not just what SolarWinds thinks I want to know. 

                               

                               

                               

                              Description: small logo

                               

                              Jeremy Mayfield

                               

                              IT Director

                               

                              4750 E County Rd. 470

                               

                              P.O. Box 445

                               

                              Sumterville, FL 33585

                               

                              Office:  (352) 569-5393

                               

                              Fax:  (352) 569-5397

                               

                              E-Mail:  <mailto:jmayfield@americancementcompany.com> jmayfield@americancementcompany.com

                               

                               

                               

                              Website:  <http://www.americancementcompany.com/> www.americancementcompany.com

                               

                               

                               

                              PRIVILEGED AND CONFIDENTIAL: This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.

                              • Re: Log Correlation: There Can Be Only One
                                familyofcrowes

                                For my company, we would like to see things like great search capabilities.  We search the logs everyday for all sorts of non-repetitive reasons.  Also, things like VPN reports, how many users, ability to get how long they were connected and when the disconnected.  What servers have the most logins, or failed logins.  Who made the most changes on the network yesterday and the ability to expand that to see what devices were affected.  The ability to remove logs from the appliance when certain data is found to be useless.

                                 

                                I find that managers what something different as often as the week changes.  Quick searches, via a simple interface similar to google/bing would answer a significant number of my customers requests by allowing them to run the queries.  The ability to export these serarches into a "report" such as a template PDF would be icing on the cake....

                            • Re: Log Correlation: There Can Be Only One
                              familyofcrowes

                              I like the analogy.  Big difference is that Highlander had a brain that could do amazing things and log correlators AI only go so far.  There is no way you can beat an experienced human being.  But...  A tool like LEM that can organize the data and look for trends is invaluable.  Rules are static and do not understand "personality".  Its the personality of the people using the applications that change the serverity of the rules dynamically, and a tool cannot understand this.

                               

                              Management is constantly looking to automate everything and reduce the ongoing cost of human resources.  But, some things take human brain power and this is one of them.  You can reduce your expenses with a tool like LEM by organizing and creating meaningful rules and filtering out many false positives, but in the end experience and a God given intelligence will rule the day.

                                • Re: Log Correlation: There Can Be Only One
                                  byrona

                                  Management is constantly looking to automate everything and reduce the ongoing cost of human resources.  But, some things take human brain power and this is one of them.  You can reduce your expenses with a tool like LEM by organizing and creating meaningful rules and filtering out many false positives, but in the end experience and a God given intelligence will rule the day.

                                   

                                  I totally have this problem as well.  Management doesn't seem to understand why I can't make alerts from the NMS "black and white" so that the alert tells them (the NOC guys) exactly what to do to fix the problem without any thinking involved.

                                   

                                  I try to use the medical analogy; the NMS is the diagnostic equipment, not the doctor.  The equipment helps gather data but the doctor needs to take that data combine it with their own knowledge and troubleshooting to ultimately make a diagnosis.

                                  • Re: Log Correlation: There Can Be Only One
                                    Mrs. Y.

                                    I actually think it's possible to automate things better. However, the problem with the Big Data guys is that they're looking at it in the wrong way. I think we need to follow an FFT model (fast and frugal tree), similar to what is used in hospitals and by first responders. But some research needs to be done to find the right elements or markers of the FFT. The problem is the probability model we're using. I believe we'd be better served by using models based on smart heuristics. I presented some research on this topic at RSA Security Conference and you can view my slides here: http://www.slideshare.net/chubirka/mash-f41-a-chubirka

                                  • Re: Log Correlation: There Can Be Only One
                                    byrona

                                    This thread actually got me to thinking about the correlation of data between the performance/event monitoring system and the log & event management system which ultimately led to a LEM feature request HERE.

                                    • Re: Log Correlation: There Can Be Only One
                                      matt.matheus

                                      I agree with your comment, and I wonder about why we aren't focusing on developing better methods of identifying important data upon collection, instead of just adding every bit to the ever-growing pile.

                                        • Re: Log Correlation: There Can Be Only One
                                          Mrs. Y.

                                          We can do it in other scientific fields. For example, there's something called DNA barcoding: http://boingboing.net/2013/05/15/the-technology-that-links-taxo.html

                                          "The non-fictional tricorder is based on an idea called DNA barcoding, which originated in 2003 with Canadian biologist Paul Hebert. He thought there might be an easy way to quickly identify species using short DNA sequences that are unique to one species or another. If you had a database of these sequences, then all you'd have to do would be to match a sample to a sequence and you'd know what species you were looking at. It's similar to the way we store fingerprints, and then use those to match prints from a crime scene with an individual person."

                                           

                                          If we can thin-slice in species identification and there's a lot more data to work with, why can't we identify markers that would be early indicators of problems?

                                        • Re: Log Correlation: There Can Be Only One
                                          Webbster

                                          I am just starting to get into the security side of the house. At this time we are using a combination of services that Dell is providing to us and Snort software.

                                          • Re: Log Correlation: There Can Be Only One
                                            michael2907

                                            This is probably one of the most interesting analogies to log data I think that I've seen. But it does have truth to it nonetheless.

                                             

                                            And you're right, if you don't spend the time upfront to properly set thresholds you'll soon find yourself in a bad mess when something happens you didn't foresee.