12 Replies Latest reply on Feb 12, 2014 10:44 PM by byrona

    !LEM Thoughts of the Week: Detecting the Target Breach?

    nicole pauls

      Hey All, Trying a new LEM idea courtesy byrona - a LEM/SIEM topic of the week. I'll try to post a topic related to security and log data once a week, you toss in your two cents based on your experience of what would/wouldn't work. If you have suggestions for future topics, toss them my way. I thought we'd start this week with something topical...

       

      The Target Breach! Over the last couple of weeks, more details have come forward about what actually happened. Check out this article from Brian Krebs with the details: Target Hackers Broke in Via HVAC Company —  Krebs on Security (check out his previous articles for the full unravel). We know from sources like the Verizon Data Breach Investigation Report (DBIR) that a lot of people aren't detecting breaches with log data, they are instead getting reports from third party sources, sometimes months later, and sometimes using the log data after the investigation is called in (other times just ending up in "we don't know, we're sorry, we'll do better, we promise"). Even when they HAVE log data being stored, it's just not being used, or being used much too late. So, what could we practically have detected?

       

      • The HVAC vendor breach: the creds to get to Target's system were stolen from the HVAC vendor. Not enough info to go on here to see if we could have detected this one.
      • Remote use of the HVAC credentials: now we're talking some basic bread and butter! (Not to mention PCI rules about monitoring remote access....)
        • Sometimes we have to rely on vendors to do remote work. Granted, maybe we shouldn't have an always-open tunnel if it's occasional maintenance, maybe those passwords should expire, and maybe we shouldn't allow it from just anywhere, but let's pretend it's a pretty regular occurrence (and it's possible since the vendor was breached they could have used normal business hours and an internal IP to get to Target).
        • Audit successful and failed usage of those accounts, PERIOD. Is this alert-worthy? Yes, if you have vendors that are logging in infrequently. If not, this is what reports are for!
        • Spot check (using reports/historical analysis or sometimes alerting if you can narrow it) logon patterns that fall outside the norm. Are they logging in at a weird time when that business isn't even working? Are they logging in when your IT team didn't even solicit their help? Are those credentials being used by some IP other than the HVAC vendor's normal IP range? Are they logging in way more frequently than usual? Are they logging on to different machines/systems than usual?
        • If you're using a pattern for account naming (like svc_* or ven_* or adm_*), set up a safety net alert/report/whatever (as long as you look at it) for those accounts being used.
      • Malware on the POS systems: could we pick this up somehow? Maybe some unusual ports being used, unusual processes being launched on those servers? Unusual traffic data on the inside or outside of the network? Unlikely we'd pick up the infection itself, assuming it was sophisticated and undetected by traditional tools.
      • Data being offloaded to external systems: here, we might have another chance.
        • We could monitor for unexpected data exfiltration behavior - if you've got a good idea of the IPs or sites that a server should be communicating with and you're egress filtering (or even just auditing egress traffic, or looking at flow data), you should be able to catch the unexpected access. For example, a WSUS server shouldn't phone home to Brazil, the MS update servers are pretty well known. Either using historical analysis or alerting that filters out the "known good" you could pick out the gray/unknown. This takes some time to figure out, just like those egress filters, though, which can make it frustrating at first.
        • Monitoring out of band network communication if you use a proxy or other web filter - someone not using your proxy/filter for FTP or web traffic might be attempting to bypass to surf some questionable sites, but they might also be trying to push data or bypass policy. Probably easier to do on servers, too, because the chances they are misconfigured is lower.
        • Spot checking (similar to that authentication traffic) download/upload patterns on servers outside of the norm, even using your proxy/webfilter. A server probably isn't going to be pushing much data up with FTP (or even HTTP) to the internet, so why's it happening?


      It's unfortunately clear at this point that some of Target's failures are evidence of PCI non-compliance, so that's kind of a step zero. Verizon just released a PCI Report that has some interesting statistics on that, we're hoping to talk about the interesting stuff in that here on Geek Speak sometime soon (maybe next week's topic? "PCI sucks, but here's how to not ****" ).


      What do you think? Is some of this stuff "DUH" level stuff, or is it just too far out of reach given the day to day task of just getting stuff done? Did I miss anything you could have detected? Anyone have interesting stories or thoughts on related topics (vendor access, evil hackers, how chip and PIN won't save the day when you're not PCI compliant in the first place)?

        • Re: LEM Thoughts of the Week: Detecting the Target Breach?
          byrona

          Well, since I had suggested this I guess it appropriate for me to be one of the first people to comment.  I actually have a lot of comments swirling around in my head regarding this so I will do my best to get them out in a coherent manner.

           

          I recently watched a webinar from a FIM vendor where it was shown how the malware would have been easily detected using some simple FIM software.  I find it scary how simple it would have been to detect something like this in so many different ways yet I think most organizations would have missed this in the same way that Target did.  I guess it's one of those things where hindsight is aways 20/20.

           

          The vendor account thing is definitely something you need to be very careful with and monitor very closely.  While it may seem like a lot of work I think it would make the most sense to trigger an alert on every use of a vendor account out to a ticketing system or run a daily report and have those logins approved.  It may seem like a bit of work but vendor accounts can be a huge exposure and need to be very closely managed.

           

          I think the using naming patterns is a great idea!  This provides the necessary meta-data to easily apply logic against in so many useful ways.  I am totally putting that idea in my back pocket for later use.

           

          I liked the example of a server phoning home to Brazil.  This is a good example where good visualizations of data would be helpful.  Just showing the IP's isn't very helpful because I doubt many people can look at an IP address and tell which country it belongs to.  Having a good visualization showing an icon representing the country would quickly grab your attention if that country were not one that server should be communicating with.  Having the data is important but having good ways to visualize it in useful ways can help you quickly identify something that isn't right or out of place.

           

          I personally think a lot of this is "DUH" stuff that is also just out of reach.  I think it's out of reach because it's very difficult to show the value in the cost to implement a SIEM and the staff to do the active management, and analysis of the data.  I don't really know how to explain it but I have found it very difficult to get people to wrap their heads around SIEM and the power that it can bring to the table.  I think when a lot of less technical people look at it they just see another log management system that costs more $$$ and don't see that value which makes it a very difficult sell and that puts it out of reach for the technical people trying to get it into place.

            • Re: LEM Thoughts of the Week: Detecting the Target Breach?
              evanr

              Well if they have to be PCI compliant they will need the SIEM implementation regardless of the costs.  It's a requirement.  I know in our environment all we have to do is say its for PCI compliance and purchasing will usually get it done.  I would agree that most of this is "DUH" level stuff.  I haven't read the article but they should be monitoring all there entry points into their network like a hawk. 

            • Re: LEM Thoughts of the Week: Detecting the Target Breach?
              nicole pauls

              It really makes you wonder, given that this kind of stuff seems obvious, what the barrier to entry is - too much other stuff that it's low priority? Even in the Verizon report people who are collecting logs frequently aren't using them. Log data itself is overwhelming, which is one reason why I decided to make the thwackCamp 2013 LEM presentation about the "now what?" that's so common in setting up LEM/SIEM, but shouldn't these be the primary reasons to use it in the first place?!

               

              Maybe "checkbox compliance" is too common - people are collecting log data, not really setting up or monitoring alerts, passing their PCI audits (somehow?) and just never touching it again. (Well, until someone reports an issue...)

               

              Or, maybe SIEM is seen like insurance, and enough people are willing to go to the ER instead of buying insurance.

                • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                  evanr

                  I would imagine its probably overwhelming to try and sift through all that log data for larger corporations.  We generally collect around 35-40 million logs a day which I'm sure is a drop in the bucket compared to most shops.  We looked at the SIEM(LEM) almost as an additional IDS tool.  Which I believe is actually in one of the market highlights on the LEM page.  I wonder how others look at it?  Is it considered just a receptacle to go back and sift through log data once the breach has occurred like you mentioned?  Is it just a compliance requirement.  Being heavily involved in the IDS realm we know how much TLC goes into getting a box set up properly.  You have to massage it and tweak it constantly.  But once done it becomes an extremely powerful tool.  We are constantly tweaking and changing our LEM box as well to adhere to our needs, and our compliance needs.  I don't think there is any excuse to just get it set up and let it sit.  We check logs daily.  It may not be fun but it needs to be done, plus its a PCI requirement Honestly I would rather be sifting through reports, log data daily than having to explain to my higher ups why we didn't catch the intrusion that may have compromised our network. 

                    • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                      byrona

                      I think it is exactly what you you have both said, it's "checkbox compliance" as well as just an archive to go back to later.  The folks that I talk to here just don't seem to see the potential power once you do the massaging and tweaking to get a SIEM firing on all cylinders.  To me SIEM is all of the following: log management, IPS, IDS, compliance, operational monitoring intelligence etc.  SIEM should be at the core of both your security architecture and your monitoring architecture; it's just as important as your anti-virus, patching, and NMS systems but somehow people don't see that.

                  • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                    byrona

                    So as this topic has caused me to look even deeper into the world of network forensics I started looking at network recording solutions that capture every packet that flows across the network.  Besides the details of the packet itself what would you loose by using LEM to capture firewall and router logs with logging turned up to the highest or at least a very high level?  It seems like this might be a more diverse solution because you still get the other capabilities of the SIEM versus a network recorder which is very single purpose.  Thoughts?

                      • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                        nicole pauls

                        You do see that solution advocated a lot by folks like Richard Bjetlich, who has the Tao of Network Security Monitoring book among others. Where it breaks down in practicality that logs have an advantage is in volume and resources. It takes more expertise to pick apart packets than it does logs (and it already requires expertise to pick apart logs), it takes a lot more storage to keep packets, and logs tend to distill information more quickly.

                         

                        There is a powerful case for them to be used in conjunction with each other, though - if you could go back to the packet trace from something suspicious in the log data, or going the other way alert from the packet data into the SIEM to be warned about potential bad behavior that logs can't see but packets can, they become Better Together. (Hopefully downstream, the stuff Rob talks about with deep packet analysis can extend in this direction - Beta for SolarWinds "Deep Traffic Analysis" now available).

                          • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                            byrona

                            I actually have the book Tao of Network Security Monitoring sitting on my desk though have actually read very little of it... its on the list.

                             

                            I took a look at that beta for DPI that you linked to; however, it didn't look at all like what I would have expected.  I guess when I am talking about packet inspection I am thinking something more like the product HERE.  The DPI beta looked more like additional features that would be included as part of NTA.

                             

                            When we turn our firewalls logging up I can see every accept and deny that takes place so the data is pretty granular.  While I can certainly see value in a Packet Inspector, it seems like you are hitting a point of diminishing returns considering the cost to have something like that in place to capture everything you could potentially need.  Maybe I need to spend some more time reading that book. 

                              • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                                nicole pauls

                                Yeah, I think the usage of DPI on the SW side is pretty new, and primarily from a network perspective, not really addressing any of the security use cases yet. Maybe someday. It would also be cool to use NTA/Flow data that way as well, kind of a first step early warning system of potential network issues that are harder to detect in the logs. (Something weird in the logs, go check the flow data for that "conversation"; something weird in the flow data, alert to the SIEM)

                                 

                                I think in the ideal cool security detect it all space, we'd have packet/flow recording/inspection/analysis, log data, asset information, threat feeds, and even system performance data, all as a data points. But you kind of have to pick your battles, and log data becomes the easiest first place to start, probably courtesy the log data being the real home for the audit type stuff. You can't really audit user logon activity via network packets, but you can with log data. So if you can get audit activity, and you can get "good enough" network activity, it is hard to justify the overhead and cost for those other advanced tools.

                                 

                                In the Target case, maybe alerting from DPI or packet capture data (heck, even netflow) could have caught the unexpected ports, protocols, and volumes of data being transferred to so many questionable sites.

                                 

                                I really LIKE the idea of packet analysis and some of the stuff he talks about in his book is really cool (or not detectable in logs), but in all practicality it might be too far down in the onion-peeling. ... especially when people are missing some basics.

                                  • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                                    byrona

                                    I really LIKE the idea of packet analysis and some of the stuff he talks about in his book is really cool (or not detectable in logs), but in all practicality it might be too far down in the onion-peeling. ... especially when people are missing some basics.

                                     

                                    I think I would have to agree, specifically when relating it back to the original topic of the Target breach.  Having SIEM and Flow Analysis in place and using them properly would have likely provided the most "bang for the buck".  Packet Inspection is great if you want to get down in the weeds and really pick apart what happened but I think it's less practical from a security/threat detection perspective.

                            • Re: !LEM Thoughts of the Week: Detecting the Target Breach?
                              nicole pauls

                              Here's an update from Krebs with some insight into the HVAC vendor: http://krebsonsecurity.com/2014/02/email-attack-on-vendor-set-up-breach-at-target/

                               

                              Spoiler: they weren't using real-time antivirus. There's more humor in the update, like using AD credentials possibly being the reason for the two networks to be connected, Target making it easy to target (HA) vendors by having some of their data public on the internet...