5 Replies Latest reply on May 11, 2012 6:38 PM by nicole pauls

    Logging Isn’t Sexy

    Mrs. Y.

      Poor little system logs; they’re the Rodney Dangerfield of the IT world, getting no respect. Log and event correlation is usually the last

      priority on the list for those at the C-level, because they need something exciting to show a board of trustees or VP. Something that

      proves the IT staff is really *doing* something. They know that centralized log correlation is important, but it’s a really hard sell.

      Trying to get people excited about log and event monitoring is like trying to sell a house based on the innovative, new wiring or plumbing.

      Most people are too busy looking at the superficial stuff: the kitchen appliances, countertops and the bathroom fixtures. They don’t really

      seem to want to know what goes on inside. Maybe if most of the SIEMs available weren’t so hard to use and implement, it wouldn’t make

      everyone tune out, especially senior management. Because according to a recent SANS survey, it looks like it’s easier to get people to focus on

      their taxes than on their system logs.

        • Re: Logging Isn’t Sexy
          byrona

          Sadly, this has been exactly my experience as well. 

           

          Now you add compliance requirements on top of that and what you have is executive management expecting you to fulfill the logging requirements with little to no budget because (as you very nicely pointed out) they don't want to spend money on things that don't "show well".

          • Re: Logging Isn’t Sexy
            cmgurley

            It would have done me well to read this thread before replying to the other, but it fits in both, so I suppose that's okay. Nutshell: I fully agree. Our network logs are a tsunami and trying to whittle them down with an SIEM seems like swallowing that wave one gulp at a time. I'm on my third day of doing so with LEM and I'm not sure it's a winning battle.

             

            If our goal is the "perfect network" (as if that actually exists) without warnings or errors (even noisy, non-impacting ones), then the logs that SIEMs and good ole syslog reveal are helpful. But the thing is, if I didn't know about most (any?) of what I'm now seeing, our network would still fly and performance would still be high.

             

            The dreaded "S" word, Security, is the exception that most might throw into this discussion, and if you're under compliance regs (like it sounds like byrona is), then there is at least some form of a backbone behind the initiative. Beyond best practices, though, parsing security logs just isn't a priority to our mgmt. We get alerts from MS SCOM when accounts lock out. We stay patched up with MS SCCM. And we keep as many ports shut as possible on our firewalls (just to mention a few BPs). With limited resources, where's the fire it would take to light up logging?

              • Re: Logging Isn’t Sexy
                byrona

                My only saving grace when it comes to the compliance requirements is that they (compliance requirements and auditors) seem less concerned with real time alerting and correlation from log data and are more interested in making sure we have the logs for forensic analysis... that is with a few exceptions.

                 

                More and more I am starting to believe that the fabled "log correlation" is much more work than it's really worth. It always looks good in demonstrations; however I have not had much luck with it in real environments, especially environments that change as frequently as ours does.

                 

                Ultimately I think that having centralized log management is important, just minus the correlation piece.  Monitoring system, application and user experience seems much more beneficial and when those indicate a problems I can go look at my logs for clues if necessary.

                  • Re: Logging Isn’t Sexy
                    Mrs. Y.

                    As a security professional and former Unix engineer, I still hold out hope for the promise of log correlation. It's the only way we can hope for real-time detection, because firewalls are not really the preventive measure people think they are. But you can read more of my thoughts on that here.

                      • Re: Logging Isn’t Sexy
                        nicole pauls

                        For a while, it looked like regulations were trending more toward real-time - especially in the federal government - but PCI took a step back fairly recently by making some of their requirements more lax on a time scale. From a security perspective, time counts because it is linked to exposure; from an operations perspective, time counts because it leads to cost savings. (We had a LEM customer that literally lost money when they got infected by a virus because their entire business - and they were in the business of taking/trading money/stock - had to stop while it was cleaned up. They could care less about the potential security breach because the network was isolated, but it was a HUGE business issue. I'm sure the networks they were trading with would care more about the security issue, though, and appreciate the shutdown/cleanup requirement )

                         

                        That said, what I see with SIEM is that most people build "correlations" that look something like "A happens with B C and D specific parameters/traits" - it's a pretty simple form of correlation (you take something you know about the traits of an event and try to pick them out of a huge quantity of data), but it's what keeps the business moving. I can't tell you how many people we have that just want to know about simple stuff like accounts getting locked out and people surfing places they shouldn't (or attempting to). The reality of the problem, as Byron is saying, is really that most people use log data to alert them of something they can't get somewhere else, then for troubleshooting and root cause analysis. That's still a huge step forward, and it starts to increase your familiarity with how log data is useful. You can also take whatever you learned in that analysis and turn it into correlation/real-time rules (if this happens again tell me), which over time adds value. It's a daunting task going into it, though.

                         

                        The Verizon Data Breach Investigation Report has said for the last few years that evidence of breaches is in the logs a vast majority of times they occur, but people don't find them until they are given another indicator (usually, customers complaining). That follows the "use logs for investigation" pattern, too, and I still think that is a HUGE step forward for a lot of organisations. Having that intelligence at your fingertips, even if you're not monitoring it in real-time, is awesome. But how do we make it easier get to the next step?