38 Replies Latest reply on Jan 30, 2014 2:08 PM by Kevin Rak

    Don't Panic and Know Where Your Logs Are

    mfmahler

      Information Security Event: An identified occurrence of a system, service or network state indicating a possible breach of information security policy or failure of safeguards, or a previously unknown situation that may be security relevant.

       

      Information Security Incident: A single or a series of unwanted or unexpected information security events that have a significant probability of compromising business operations and threatening information security.


      -- ISO 27001 Standard

       

       

      It was Sunday of a long weekend and you were enjoying a short break from work. You were feeling good that you would also have a day off the next day. All the sudden, you received an high-priority email on your iPhone. Your IT Director forwarded you and your team a complaint from another company that they received inappropriate email, which was sourced from your public IP address. The IT Director wanted you to look into it and to get back to him.

       

      No more time off. You VPN'ed to the corporate. You made a few phone calls and pulled in resources from different departments. Your network engineer confirmed the public IP was the NAT IP of half of the company's outbound traffic, BUT not of the email gateway! "Great", you thought, "We allowed port 25 outbound from internal? Did we turn on outbound logging on port 25?" Bingo. Your firewall engineer told you that it was logged. What else? You had the destination IP, so you asked your WAN engineer to run a NetFlow report.

       

      Fortunately you had all the information you need to pinpoint the infected workstation. You had to beg the bad-luck field engineer to clean up the workstation in the long weekend by himself. Wait a minute. You wondered if you had enough logging information to determine how the workstation had been infected. Your brain had been spinning for the postmortem report. Now you know that enough logging may not be enough.

       

      Do you find the story familiar? Did you experience something similar at work? How does your organization handle incidents? Did you happen to deal with the law enforcement?

       

      And One More Thing: Bad guys know when will be the next major holiday and only the junior InfoSec personnel in the office.

        • Re: Don't Panic and Know Where Your Logs Are
          syldra

          Do you find the story familiar? Did you experience something similar at work?

          yes, I've had something like this happen on a few occasions and at different jobs. From relaying turned on on a mail server to being used as a storage center for warez...

           

          How does your organization handle incidents?

          Analyze -> identify -> mitigate -> resolve -> analyze, documenting each steps if relevant (usually is)... Depending on the causes and mitigation/solution, it falls to different teams to take the actions needed

           

          Did you happen to deal with the law enforcement?

          I'm glad I did not. I feel it brings another level of complexity when things go legal...

            • Re: Don't Panic and Know Where Your Logs Are
              zackm

              syldra wrote:

              How does your organization handle incidents?

              Analyze -> identify -> mitigate -> resolve -> analyze, documenting each steps if relevant (usually is)... Depending on the causes and mitigation/solution, it falls to different teams to take the actions needed

               

              Luckily, I don't have incident response in my job description anymore. However, when I did, this was exactly what we did.

               

              As far as law enforcement, not really. We did have to prepare logs for the legal department a few times, but they never actually used them for anything from what I remember.

              • Re: Don't Panic and Know Where Your Logs Are
                mfmahler

                Well put syldra !!!

                 

                I think I should have mentioned that the story was based on a real life event.

                 

                In my organization, incident handling is managed by the Information Security Office and the Network Security Team (my team) provides technical assistance. The InfoSec Office enforces the policies/procedures and my team sets up the infrastructure to collect logs.

                 

                Thankfully the InfoSec Office dealt with the law enforcement. Yeah, I'm glad that you did not have to, too.

              • Re: Don't Panic and Know Where Your Logs Are
                mbwalker

                Now there's a really cool frood who knows where his logs are!

                • Re: Don't Panic and Know Where Your Logs Are
                  supermon

                  syldra summed out our response:

                   

                  Analyze -> identify -> mitigate -> resolve -> analyze, documenting each steps if relevant (usually is)... Depending on the causes and mitigation/solution, it falls to different teams to take the actions needed

                   

                  I'm not personally involved with the process unless logs are needed.

                  • Re: Don't Panic and Know Where Your Logs Are
                    cahunt

                    syldra hit the Round Nail Head Square! And will get good points for the mentions. Who can argue with...

                    Analyze -> identify -> mitigate -> resolve -> analyze, document  ---- I will add, PREVENT... if anything can be done to better secure or stop this from happening again.

                     

                    I often get the what happened here... and the times where log's are not enough is 'sometimes' ... Netflow in full is coming, More Granular Monitoring is also in the works to allow for more data to try and prevent the lack of data.

                     

                    As far as Law Enforcement... No informers. I know the law's and adhere; Any infraction or issue of legality i take up the chain, documented. To someone who makes those decisions; and then they are responsible...   So when the fan hits the tihs I am not actionable, legally or monetarily.

                      • Re: Don't Panic and Know Where Your Logs Are
                        syldra

                        Agreed cahunt, preventing is also part of the process, sometimes the final resolution readily prevents further problems, and sometimes the second round of analysis brings to the front things we had overlooked. The continuous documentation process also serves this purpose, more or less.

                        • Re: Don't Panic and Know Where Your Logs Are
                          mfmahler

                          cahunt It looks like your organization is improving on visibility and can give you guys better picture of what's going on.

                           

                          Interestingly enough that NetFlow was not originally created with security in mind, but over the years, vendors promoted it as a information security tool. What can be better when you have a full view of packets/flows in the network? NetFlow tools always have canned top-talkers report. With that tools can build baseline and from that can alert any abnormality.

                           

                          The security folks always prefer Full NetFlow. However, when the advancement of 40G/100G bandwidth in the data center switches, there is a trend to move to Sampled NetFlow. I was told it's due to the ASIC limit for 40G/100G traffic.

                            • Re: Don't Panic and Know Where Your Logs Are
                              cahunt

                              Our new tool was a combination of engineering and Information Security. The tool as robust as it is can be troublesome in the hands of the wrong analyst.

                              Our Border/Edge traffic is a main component of what they will be watching and sampling with this new tool, this is a good thing.

                               

                              Really, I want access and the ability to at least request some customization so that I can import the really important parts that we care about into a few Orion views, specifically the NOC type setup we have.

                          • Re: Don't Panic and Know Where Your Logs Are
                            Aaron Denning

                            Agreed syldra i had to deal with this when i worked Information Assurance getting calls all the time had a special ringtone for the office just so i knew when it was about to be a bad weekend. never had to get the law though thank goodness.

                            • Re: Don't Panic and Know Where Your Logs Are
                              wbrown

                              Luckily I've only had to deal with that once in a previous lifetime as a VAR.

                              We managed the firewall and network for a client and the request simply came to me as "an email appears to have been sent directly from a workstation on date/time X.  Confirm or refute that claim and if confirmed tell us what workstation that was."  As the client was a law firm I had a suspicion that everything I state better be independently verifiable and contain nothing but fact - no opinion or speculation.  I was also asked to personally hand the findings directly to the owner in a sealed envelope.

                              Never did find out what was sent or what the fallout was but I think it's safe to say somebody's job was affected.

                               

                              I like having a dedicated InfoSec department to handle that stuff.  My current employer is large enough that some such investigation (infected host, malicious traffic, etc) can potentially take place daily.  Like working on a puzzle it can be interesting to sort through logs to go from point A to Z and figure out that puzzle, but I don't imagine it to be much fun when that is a regular part of my job.

                                • Re: Don't Panic and Know Where Your Logs Are
                                  mfmahler

                                  wbrown Thank you sharing your past experiences. In my current lifetime at work, when I am not building or troubleshooting the data center network, you'll find me dealing with network security issues.

                                   

                                  My team and I had similar experiences of sealed envelope. Sometimes we needed to provide web browsing evidence to the Human Resources. You probably can guess what were those outcomes.

                                • Re: Don't Panic and Know Where Your Logs Are
                                  matt.matheus

                                  We have a dedicated information protection department that handles these types of events.  My team may get a request to look at network traffic, but not much more.  The information protection department handles all logs and event monitoring with a centralized appliance which all other departments send logs to. 

                                  • Re: Don't Panic and Know Where Your Logs Are
                                    Radioteacher

                                    Fortunately, we have not had such an incident in many years....so long ago that we did not the current tools in place.

                                     

                                    Last one that comes to mind is Nimda.

                                     

                                    Nimda - Wikipedia, the free encyclopedia

                                     

                                    RT

                                      • Re: Don't Panic and Know Where Your Logs Are
                                        mfmahler

                                        Radioteacher Wow, your last incident was Nimda. That's incredible! Your organization certain has some nice tools and defense implementations.

                                         

                                        You brought up my memory. Many years ago I joined my current network security team and since then we have been expanded in manpower and have been improving on tools, defense mechanisms, policies, etc. I was told that before I had joined the team, every year in August there had been computer virus outbreak.

                                      • Re: Don't Panic and Know Where Your Logs Are
                                        russb

                                        Given that these logs DO contain pertinent information, how do you monitor and alert on them 24 x 7, rather than on demand following an incident?  Can't do it manually as there is no budget for someone to do this full time (at least not in the Education sector).

                                        • Re: Don't Panic and Know Where Your Logs Are
                                          michael stump

                                          I've been through many security incidents like the one you describe. I have mixed feelings about incidents like this, though. First and foremost, I lament the lack of security controls (specifically monitoring and logging) in place at most shops. They'd prevent, or at least minimize the impact of a break-in. But honestly, I kinda like the challenge of tracking the attack from the point of entry. It's an intellectual exercise to determine how they got in, what sploits were used, and how the attack traversed the infrastructure. It's a chance to be creative and to pull data from multiple sources in order to document the attack, which leads to the infrastructure and policy improvements to prevent future attacks.

                                           

                                          I'll say again that security in general, and logging specifically, has to be treated as an evolving practice. You can log the hell out of your infrastructure and still not capture everything. So when you have an event that exposes a gap in your auditing, you correct it. If you have multiple events that exploit the same vulnerability, you've squandered your golden opportunity to learn from your mistakes.

                                           

                                          Oh and one more thing: NEVER clean an infected workstation. Get rid of it, or if the incident requires it, establish a chain of custody and allow law enforcement to take ownership of it. Truly insidious malware can't be cleaned. Best to start from scratch.

                                            • Re: Don't Panic and Know Where Your Logs Are
                                              mfmahler

                                              _stump You are absolutely correct.

                                               

                                              Oh and one more thing: NEVER clean an infected workstation. Get rid of it, or if the incident requires it, establish a chain of custody and allow law enforcement to take ownership of it. Truly insidious malware can't be cleaned. Best to start from scratch.

                                               

                                              In an Incident Handling class I learnt "Game Over": once a computer is infected, no matter it's a computer at home, a workstation at work, a server in the data center, it's game over. I wonder how many organization actually follow this best practice.

                                                • Re: Don't Panic and Know Where Your Logs Are
                                                  byrona

                                                  mfmahler I really like the "Game Over" concept but I am not sure how well that plays out in reality.  As a use case: if one of our customers systems became infected and I followed the "Game Over" strategy, I would need to delete that Virtual Machine.  Before deleting the system I would need to migrate all of the customers data; however, that could potentially risk compromising the new system if any of that data wasn't clean.  I guess the only scenario where this might work if you were able to pinpoint the time of corruption and had a data backup from before that time that the customer could use though this would only work if you didn't have to go back very far.

                                                    • Re: Don't Panic and Know Where Your Logs Are
                                                      mfmahler

                                                      byrona For user workstations we connected the infected workstation from the network and removed the hard drive for for forensics. The workstation would be rebuilt. Logs and SIEM came into picture to determine how the workstation had been compromised. There is also a practice that we don't implement. When a infected workstation is discovered, it is taken off from the network but is no turned off. All the forensics would be performed on the live system.

                                                  • Re: Don't Panic and Know Where Your Logs Are
                                                    byrona

                                                    _stump What tools do you generally use for your log management?  I saw in a different thread that you like Snort, it sounds like you have a lot experience with this type of thing so I am curious what you have in your log management toolbox?

                                                      • Re: Don't Panic and Know Where Your Logs Are
                                                        michael stump

                                                        I do like Snort, but it's partially for nostalgic reasons. I tend to favor tools that don't get in the way of the job to be done. Snort is not user friendly, that's for certain. But once you get it running, it's out of the way and you can focus on your data. I prefer text logs instead of database-centric tools, because I find that grepping through logs is much faster when I'm looking for something specific. Again, it's an old habit that I haven't seen a compelling reason to change.

                                                         

                                                        For log management, I rely on standards for Linux and UNIX platforms like rsyslog and logrotate. I've heard good things about syslog-ng, but personally I find it easier to roll out rsyslog. swatch is good, too, for pattern-matching and log scraping.

                                                         

                                                        How about you? I'm always keen to learn about how others manage their logged events.

                                                          • Re: Don't Panic and Know Where Your Logs Are
                                                            byrona

                                                            I am a Windows guy so my options are a bit more limited since Microsoft seems to hate the concept of logs.  Where I work we have both Linux and Windows and our syslog server has always been a syslog-ng system and it works great for just taking in a large quantity of logs for digging through later if need be.  Recently we I started working with SolarWinds LEM and while it certainly has a learning curve and a few things that could use improvement, overall I find that it works pretty well and requires very little management overhead as it's a virtual appliance.

                                                      • Re: Don't Panic and Know Where Your Logs Are
                                                        Kevin Rak

                                                        We had a similar situation recently. Despite not being able to call on different departments to do each task, as described above, I was able to look at the traffic logs, find the source computer, and clean it from our anti-virus console, without even interrupting the user's work-flow. Their computer was also running XP so with the upcoming refresh we have planned to Win7 I also switched out his pc for good measure.