57 Replies Latest reply on Dec 10, 2014 8:31 PM by jkump

    To Log Or Not To Log: That Is The Question

    mfmahler

      "SNMP is vital. Syslog saves life." -- a networking guy

       

       

      Hello Thwack, I'm Gideon Tam, one of the Thwack Ambassadors for the month of January. Happy New Year 2014! I'm a network security professional and also a data center network architect.

       

      When you ask the Information Security folks what they think should be logged, they probably give you a simple answer: EVERYTHING. But in reality, things are more complicated than this.

       

      Did you encounter Windows Server Admin folks who refuse to install any log connector agent on their servers because of their bad-taste -in-the-month experiences with foreign agents? They may not be even willing to give out admin credentials for agent-less log pulling. This situation sometime is resolved when the Information Security Officer tells them that they are making their career decision.

       

      IIS 6.0 provides six different log file formats. IIS 8.5 now provides enhanced logging. What to log and what not to log becomes something that needs to be worked out between Application and InfoSec groups within a organization.

       

      Detailed logging on firewalls and VPNs is a common sense. But what about logging DNS? Do you want to endure the high volume DNS logs for event correlation? Decision. I saw organization just simply logged DNS on devices and rolled over due to the volume. It didn't mean it's bad. It's just based on the face that wether they would use the data.

       

      Networking devices have different level of logging. What level to log all depends on the need. The lower level doesn't give you much information and the higher level adds device's CPU load.

       

      To do it right, logging requires collaboration of different departments within an organization. Sometimes this area is not in high priority, which it should be, in an organization. Do you agree? Did you experience similar situations? Did you have any conquering story?

        • Re: To Log Or Not To Log: That Is The Question
          Charles Galler

          I have administered a pair of ASAs for an organization with 4000+ employees. The security group asked me to send them the ASA logs at the highest level. The ASAs could support it, but I argued with them that they didn't want the debug level and finally talked them into getting informational level. After I configured it to send that, they asked for me to turn it down even more.

           

          The next question is that you have all of that logging data being collected, who is going to dig through it or monitor it to really see what is going on? How do you find the needle in the haystack to find the bad behaviors or intruders in the network?

          • Re: To Log Or Not To Log: That Is The Question
            byrona

            I think you need to start with a plan.  What do you want to use your log files for, what rules do you plan implement on your logging system, how do you plan to audit your logs, what do you plan to audit your logs for, what operational rules do you plan to put into place with regard to log management, and ultimately what is the problem you are trying to solve by implementing a log management system in the first place.  Once you begin to answer these types of questions you can begin to formulate a plan and decide what logs you actually need.

             

            I just read an very interesting blog on how Etsy is handling something very similar (thought not exactly the same); I found it very interesting so if interested you can read that HERE.

            • Re: To Log Or Not To Log: That Is The Question
              zackm

              While I really hate to admit it, one of the most seamless logging situations I have encountered was when a CIO told the entire IT staff of a nationwide company that the security officer had the final say on logging. Period.

               

              At some point, someone has to be given the reigns to make the final decisions on architecture. However, that someone also needs to be a humble person who takes the advise of the SMEs within each group they interact with. That's a hard person to find, and a hard position to fill as everyone always hates the security officer   (something I always say "when the security team is doing their job, you cannot do yours) Of course, this is all in good fun and anyone who has spent a day in our industry knows the dangers of ignoring security.

               

              I would HIGHLY agree with cgaller here as well. Logging is all well and good. But meaningless data stays meaningless until someone parses through and analyzes it.

              • Re: To Log Or Not To Log: That Is The Question
                supermon

                As byrona said, a plan is critical.I find the best way to get the buy in from other teams is sell them on the abilities and how it can help them with troubleshooting and early detection. There are times when we find a problem, but do not have the logs to actually determine root cause and how to prevent a re occurrence.

                And sometimes its a matter of selling the setup to someone above the guys resisting.

                • Re: To Log Or Not To Log: That Is The Question
                  cahunt

                  I get a lot of, 'tell us what happened' ; so for me logs can be everything when your dealing with the aftermath. The correlation of logs with events allows for a full picture of the events that occurred. Of course the meaningless data just clutters the page when you are trying to find that needle. Having someone who is unfamiliar with the data can be fun; as they like to scream anytime they see red. We have certain analysts with Security that kick up all sorts of things when on call.

                  zackm is right when someone like that is doing their job it really cuts into your day.

                    • Re: To Log Or Not To Log: That Is The Question
                      Charles Galler

                      The shows #173 & #174 on the Packet Pushers podcasts have some war stories about CSOs who ran around with their hair on fire saying the companies were hacked. The real root causes were often human error and took time to locate.

                        • Re: To Log Or Not To Log: That Is The Question
                          cahunt

                          I often get ticket's with minimal info asking to track down a device; or provide a location or what user is on the device.

                          Wireless is easy(NCS/Prime), but the wired connections can be fun, especially starting with only an IP and the time of attack, or bad traffic.

                           

                          This has come to be a specialty of mine so everyone thinks..I get almost every ticket for a lost machine, or malicious hiding device.

                        • Re: To Log Or Not To Log: That Is The Question
                          mfmahler

                          cahunt I got that a lot, too. If logs are everything when dealing with aftermath, did you have story like, "oh, I guess we send logs from this or that device!"?

                            • Re: To Log Or Not To Log: That Is The Question
                              cahunt

                              My logging server is apart of our templates; the hard part was when we upgraded a long while back and moved the servers to a different data center was getting not only the old/current nodes updated (NCM Job) but was getting the updated templates to everyone for use. I still find references to our old or older logging servers and have to blow them out. Anything I setup I make sure it reports to NPM; and the firewalls that I monitor specifically are set to send almost anything to traps (mine FW's don't change ever once in place, very specific app) - I do not set the logging levels for other FW's.

                               

                              I use the logs to tell the story of why something went down, or how it did happen.... or in some cases why an alert triggered when the services were still operational.

                              for having no, single point of failure - we monitor every node by a single IP, and must manually confirm when we see a down and users still connected or services and traffic still working and flowing.  I would like to hear some idea's and tactics on removing the single point of failure from the monitoring. (MGMT IP, and a Gateway or two, using Orion + another system to monitor, etc.)

                                • Re: To Log Or Not To Log: That Is The Question
                                  mfmahler

                                  cahunt In my organization, we have enterprise-class monitoring system that monitors all IT devices, servers, routers, switches, firewalls, VPNs, room temperature, etc. For networking and security devices, we send syslogs to centralized syslog server and then feeds to SIEM. We also have other sources to send to the SIEM. People who make SIEM know how to make their living by charging per incoming source. We save our organization's money by collecting sources in a couple of place and then feeding the SIEM.

                                   

                                  Majority of the networking and security devices send SNMP traps to the enterprise monitoring system so that we can be alerted for the device health. We also utilize NetFlow to give us performance measure and security analysis.

                            • Re: To Log Or Not To Log: That Is The Question
                              ecklerwr1

                              Templates for NISPOM compliance would nice... it's a moving target to a degree as it evolves a little year to year but I suppose all do whether it be SOX, PCI, DSS or what.

                              • Re: To Log Or Not To Log: That Is The Question
                                michael stump

                                Agreed that you need to work with all interested parties to determine logging level. Logging at debug for everything will give you great information to use in the event of a security incident. But the overhead required to generate this data, and the storage required to house the data, may be too costly for the benefit that debug provides you.

                                 

                                Yes, to invoke debug level logging is a bit heavy-handed. But you get my point.

                                 

                                Your best bet is to document your logging levels, and clearly explain what is and what is NOT being collected at the current log level. Then get everyone to sign off. If management wants more logging, then build out the business case for it and show the true costs of generating and storing non-trivial amounts of data. Let management make the call on whether the investment to create a robust logging infrastructure is worthwhile.

                                  • Re: To Log Or Not To Log: That Is The Question
                                    mfmahler

                                    _stump Wow!!! Wow!!! I really like your suggestion of documentation policy. With that, everybody in every team is on the same page and has the same expectation. You know, when incident hits, sometimes, we, people may not be able to think straight; finger-pointing aftermath is not uncommon. Documenting the expectation and the reality also gives us a reference point to improve upon if we "screw up".

                                     

                                    Do you have any joyful or upset story in this area?

                                  • Re: To Log Or Not To Log: That Is The Question
                                    matt.matheus

                                    I think you have to start with an open conversation with the security team.  What are they trying to accomplish versus what your systems can provide.  Too many sec officers want every possible log, but then are surprised with the deluge of information... most of which isn't meaningful to security.  If the security team can be honest about what they would like to have, it makes it easier for the engineering team to provide it.  My main problem at my current job is a non-technical security officer, who understands compliance and laws, but not the technical side.  This often leads to unreasonable or impossible requests which turn into awful one-off configurations or kludged together solutions.

                                    • Re: To Log Or Not To Log: That Is The Question
                                      Radioteacher

                                      Retention and logging levels are really issues for me.  Lucky for me I have a really good relationship with the Network Security Officer.  So we are working it out together.

                                       

                                      RT

                                      • Re: To Log Or Not To Log: That Is The Question
                                        Jfrazier

                                        Logging...a subject that can be a pain but when troubleshooting is a requirement.  I understand infosec's requirement for logging and I understand SARBOX and various other audit requirements for logging.  On the other hand we also monitor log files for specific events elated to application health and system health.  In some cases it is pretty simple and we just use the built in tools to pull off what we need.

                                         

                                        In other cases there are some very chatty log files and trying to find the needle in the haystack is more the norm.  Thus we are working on a hybrid solution so that infosec gets what they need and we get a smaller firehose to drink from by using a tool design specifically for this.  Granted Kiwi is good but doesn't support some of the log file naming challenges we face.  So we are using Splunk to pull the logfiles and sending it all to infosec while sending what we need our way.  Thus we are putting only one agent on a system and everyone including the application support folks get a Win Win on being able to see things.  We get reduced overhead but all the data aggregated in a way that everyone gets to see their slice of the pie.

                                        • Re: To Log Or Not To Log: That Is The Question
                                          Kurt H

                                          Logging is a very sore subject. It seems that everyone and their brother wants access to view logs. They want logs sent to everyone so they can get reviewed on a daily basis. The biggest problem is the manpower it takes to conduct log review on a continual basis. Then the Secuirty sections want to make sure that you are viewing your logs on a daily basis to figure out what is going on. I like keeping my logs simple, showing critical and Warning logs sent directly to my e-mail That way I do not have to go through all the informational log items.

                                           

                                          I really think that logging should be done, but not on everything that goes on with a system. There should only be critical and warning items logged, not information items.

                                          • Re: To Log Or Not To Log: That Is The Question
                                            Mark Doering

                                            Absolutely! Logs are vital to gaining insight into what is happening on your systems and spotting potential issues before they snowball.  It's easier to resolve an issue caught through regular analysis of log files than when the pressure is on because it's reached critical mass and is causing a production impact.  We have fairly verbose logging on all of our servers and other devices and many methods of collection.  We use RSA EnVision for long-term archival/search (though I'd replace it in a heartbeat if I had funding), and custom scripts that ship logs into different queues (separated by environment e.g. Apps, Data, Infrastructure, Network, etc) that undergo daily manual review by our engineers (not just NOC staff, but the people who designed the systems.)  The results are then summarized with any anomalies/exceptions noted, signed off on by the engineer and reviewed/signed off by management, then stored for further review/archival/compliance.

                                             

                                            Your logs are your environment's autobiography. To understand the "now" you must first understand the "how"

                                            • Re: To Log Or Not To Log: That Is The Question
                                              uidzer0

                                              My feeling is that enough logging should be captured to meet the needs of the organization, and not a bit more.  Accomplishing such a feat is easier said than done.  Many organizations have IT departments comprised of teams who ultimately are responsible for delivering a successful IT experience to the end user.  Yet, each believe they are only responsible for their piece of the pie.  Most don't realize our deliverable goods to users is a sum of its parts and any deficiency will be reflected.  Communications exist only when problems occur.   All operate by C.Y.A. and all believe  they are the lynch pin holding it all together.  Sad, but often the case. 

                                               

                                              ITIL has been mentioned and should often be an company goal, just as ISO is for manufacturing.

                                               

                                              With a primary goal, define the war fronts.   Enhance Security.  Provide Auditing.  Ensure Compliance.   Asking teams to take on such daunting goals would likely lead a results of logs not very useful.  Instead, let teams keep work loads as they are and comprise a smaller poly skilled team comprised of enough technical know how to understand the infrastructure, grasp the equipment to know what changes are necessary, understand applications and its deployed technologies.

                                               

                                              Take auditing for this example.  Define a small hill to conquer.   - Provide the organization the ability to track user activities throughout the network

                                               

                                              Quick work plan:

                                              All teams verify NTP configuration and successful deployment.

                                              Active Directory Team - Record security logon and logoff events for all users and report to designated location.  When reporting, provide username, source machine ip address, source machine name, target ip address, target name, time of activity.

                                              Unix team, if application - Perform the same functions and reporting above.

                                              Network team - Verify LLDP or CDP enabled. Record the following access port information - Mac, Ip Address, switch ip, switch mac, time stamp, report each 5 minutes.

                                              Network team - Report each 5 minutes mac addresses associated to uplink trunks with time stamp

                                              Network team - WAN ports - Report mac, Ip address, firewall ip, firewall mac, timestamp

                                              Desktop Team - Report Login and Logoff events for users:  record username,  user sid, time stamp

                                              Desktop Team - Report Screensaver invocation and dismissal: Record username, user sid, time stamp, event id.

                                               

                                              Anyhow, you get the point.  The directive must come from the top.  But the implementation is key to the success.  Having singular, compartmentalized teams creating definitions to satisfy this small hill would be a disaster to formalize into a good report.  And if it was "successful" would their way match the goals of the grand plan.  Would each report of data or metric match necessary indexes for later aggregation and reporting.

                                              • Re: To Log Or Not To Log: That Is The Question
                                                802jr

                                                I am coming in a little later on this and have not read what everyone else is saying, Yet, this is what I have encountered InfoSec does want everything logged, I believe to cover their behinds. i am not saying that is bad, but they really never do anything with all the logs they collect. Once in a while when something goes wrong they use it to help find the cause or to put blame, that too is fine since the logs are their to record what has happened. I think that those that need/want the logging to take place should use them to constantly improve the systems they are logging. If there are a few logs that probably means that the system is running as designed and only pumping out informational messages.

                                                 

                                                My two cents...

                                                  • Re: To Log Or Not To Log: That Is The Question
                                                    mfmahler

                                                    Hey 802jr , never too late to join this interesting discussion (at least I think it's interesting ). When nothing went wrong (that is no incident), the logs we collected would seem no usage. I guess we prepare for the bad day. If things hit (and also if law enforcement is involved), those normal, common, as-if-useless logs will become valuable.

                                                  • Re: To Log Or Not To Log: That Is The Question
                                                    wbrown

                                                    Something I've heard mentioned in a few classes over the years is "If you're not going to review the logs, then don't bother logging."

                                                    So, in some smaller organizations I've completely disabled logging. Beats having to support another server just to handle and store those logs.

                                                    Once I starting admining firewalls I actually got interested in looking at those logs and realized the value of Perl.  Also learned the value of having an unofficial server sitting somewhere I could put linux on and pack in a bunch of cheap disks.

                                                     

                                                    These days I've got a security department that wants those logs and has the resources to keep them and sift through them so I don't really care what my logging levels are - I just set them to what they want and let them worry about the rest.  The nice part of this is that there is something to look through later when I'm trying to diagnose an issue.

                                                      • Re: To Log Or Not To Log: That Is The Question
                                                        mfmahler

                                                        wbrown You hit the right point! Wether log or not log, and how much, depends on many factors and resources. Different companies have different policies and implementations. These policies and implementations evolve in a better way, IMHO, especially after bad incidents.

                                                         

                                                        If the resources allow, always log. Yup, we may not even see those logs are used (may I say it's a good thing?).

                                                      • Re: To Log Or Not To Log: That Is The Question
                                                        Radioteacher

                                                        We log many ways using NPM and Kiwi but for research we use LEM from Solarwinds.  We have used LEM to solve some pesky MS AD issues when users get locked out.

                                                         

                                                        RT

                                                        • Re: To Log Or Not To Log: That Is The Question
                                                          Kevin Rak

                                                          Unfortunately, we don't really have the man-power to look through the logs that we do record - which isn't much. We occasionally glance through the Windows logs on our Windows Gateway server and at the logs on Exchange, but we haven't had the opportunity to even setup much log collection.

                                                          • Re: To Log Or Not To Log: That Is The Question
                                                            jkump

                                                            Looking at alternatives to having to manually review logs, an automated tool can be helpful.  Then, it becomes a real cat and mouse game for the amount of logging and detail information to collect.  Every situation results in how much correlation is required and how much benefit in trying chase down the rabbit.