cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Deep Packet Inspection - Security functions and Limitations

Level 10

In my last post I talked (briefly) about Protocol Anomaly Detection (PAD) and Signatures. My original intention was to talk about other topics, but given that the security aspect has been so popular, I'm going talk more about the security aspects and (quickly) circle back to optimisation and performance management later.



Obfuscation


Like any security inspection technology, there are limitations and trade-offs to Deep Protocol Inspection (DPI). DPI can only make security decisions on visible traffic; and that is becoming an increasingly common problem. Encrypting traffic with SSL or TLS was traditionally reserved for banking applications and the like, but the exploits of the NSA has lead many organisations to adopt a “encrypt everything” approach, including simple web traffic. Applying PAD or Protocol Signature checks to network flows after they have been encrypted is difficult, but not impossible. The size of the problem it is influenced by the direction of traffic you are trying to inspect. For example:


  • To inspect your organisation's inbound traffic (e.g. heading towards a public web server).  For a server under your administrative control you should have access to the private key used for encryption. In this case, the firewall (or other network security device) can decrypt the traffic flows on the fly and make a decision based on the payload. Alternatively, if you have server load balancer that supports SSL offload (and can’t think of any that don’t), you may choose to inspect the traffic after it has been successfully decrypted.
  • Inspecting traffic as it leaves your network (e.g. headed to a 3rd party website) is trickier.  When the firewall sees a client-to-server SSL session being established (after the TCP 3-Way), the firewall takes over and acts as a “man in the middle” (MITM).  The client establishes an SSL session to the firewall, and in turn the firewall creates a separate session to the outside server. This allows the firewall (or other secure-proxy device) to inspect traffic transiting the network looking for either malicious or undesirable traffic (such as credit card numbers heading out to the network). The drawback is that this method totally breaks the browser trust model (well breaks it even more). When the SSL session is set up, the server proves its identity by sending a public key; a certificate counter-signed by a Certificate Authority known (or trusted) by the browser. This mechanism is designed to prevent MITM attacks from happening. However, when the firewall intercepts the session, the user’s browser would spot the mismatch between the identity of the real server and the one provided by the Firewall. Encryption will still work, but the user will get a very visible and very necessary “nag” screen. The trick to preventing the “nag”, is to mess with the trust model further by creating a “wildcard” certificate. This allows the firewall to impersonate any SSL enabled site. For this to work, the CA key for this bogus certificate is placed in the “Trusted Root CA Authorities” list on any device that needs to connect through your network inspection point (firewall, web proxy, etc.). If your network consists of Windows 2000->8.x domain-member workstations this is about 5 minutes work. If you have almost anything else connected, it can be a significant logistical exercise.  In fact, a significant part of the challenges around “Bring your own Device” (BYOD) policies are around establishing client-to-network trust; and most of them have to mess about using certificates to do it.


Performance


As has been mentioned by several thread commenters, applying these advanced protocol-level defences have an impact on performance.

  • PAD features are relatively lightweight and are often implemented in hardware as they are dealing with a limited selection of parameters. There is only a limited definition of what the protocol-compliant service should be expected to deal with and match traffic as “good” or "bad”.  I would expect basic PAD features to be enabled on almost all firewalls.
  • Protocol Pattern matching is a bit more difficult. For each packet that comes in, it has to be matched to a potentially huge list of “known bad” stuff before it can classed as good. For common services such as HTTP, there are many thousands of signatures that have to be processed, even when the traffic has been successfully identified.  Inescapably, this takes time and processor cycles. A few firewall vendors use specialized ASICS to perform the inspection (fast but costly) but most use conventional x86 processor designs and take the hit. This is why having an appropriately and updated suite of patterns attached to your security policy is critical. Telling a firewall (or other security device) to match against all known vulnerabilities is a fools errand; it is far better to match against the most recent vulnerabilities and the ones that are specific to your environment. For example, inspecting traffic heading to an Apache server looking for IIS vulnerabilities wastes processing resources and increases latency.
  • SSL inspection. SSL inspection creates a significant workload on firewall or secure web proxy, as a result many hardware devices use dedicated SSL processors to perform the encrypt/decrypt functions. Any given device has a finite capacity, which makes it critical to decide up-front what, and how much traffic of encrypted you want to inspect.


A “good” firewall is the one that is properly matched to your environment, and properly maintained. All the fancy application identification and protocol security techniques won’t help if under production loads it barfs the first time you turn them on, or you fail to regularly review the your policy.


In my next and final post, I shall touch on the performance management and troubleshooting aspects of DPI, Thanks for reading and thank you even more to those who participated in the little survey at the end my last post!




44 Comments
Level 11

Interesting. glenkemp So, is this the same as the concept behind using SIEM (i.e. baselining a system and then watching for anything outside of the norm [anomalies, that is])?

Level 21

Good question jaimeaux!  Anomaly based detection is becoming more and more important as keeping up with signatures is slow, workload intensive and keeps you always one step behind the zero day threats.  I personally want to see more evolution in anomaly based detection for everything monitoring related.

Level 10

Hi, although I've only played with a SIEM a bit, (logs, yawn!) I believe that "anomaly detection" is quite different in the context of Firewall DPI.  As I understand it, SIEM systems are looking for usual traffic flows or volumes that deviate from the "norm". The definition of "norm" being built up over a period of time with perhaps some factory-shipped behavioural indicators.  SIEM uses anomaly detection an indicator of various "bad" things such as sudden load increases, virus outbreaks or even attacks. In the context of Firewall DPI, PAD is looking for things that deviate from the known-good; i.e. HTTP traffic should have a very specific set of behaviours as laid out in the RFC.  From what I've seen, most firewalls aren't "smart" enough to pull off the long-term analysis that SIEM can do. Whilst some vendors have their own SIEM or SIEM-like products, they tend to be external appliances. Also, PAD on a firewall is very much active and in the flow path and can react/block at point of discovery.  SIEM on the other hand is very much after then event, and whilst you can set up alert/triggers than perform some action, the "bad packet" will probably have already been delivered to it's destination.

Level 11

So from what I gather from your reply:

SIEM is reactive, PAD is more proactive (still reactive though); SIEM is external, server-based, on-going process in which it logs your network -- PAD is a firewall-based, process based less on the individual network and more on networking standards... Yes?

Level 11

Good post.

Level 11

Thanks for this article!

Your comment: "most use conventional x86 processor designs" has me going back to the manuals and finding out exactly what processor architectures our firewalls are using for inspection.

Level 9

Really good post. Thanks for sharing.

Level 12

Thanks for sharing this great article.

Level 12

great article cant wait to read more.

Level 11

Thanks again

Level 13

I have lots of things that are half full! From many viewpoints! http://crouchhouse.net/GlassHalf.html

Level 17

Encryption and Certificate Authority are going to be key in the post NSA Inspection Era.  Very nice read of features and possibilities. The idea's are flowing, and I think at this point we are drooling for a few Real World implementation solutions.

Level 11

Agreed; Real world implementations -- let's see 'em!

Level 14

Enjoying the security posts!

Level 10

Yes, I'd say that was a pretty good explanation..There are some vendors that have SIEM-like features on box, but inevitably they are going to be in a trade-off of SIEM features vs. performance vs. cost.

Level 10

Thanks! Yeah processor architectures is a really interesting topic I'd love to spend a bunch of time researching and writing about the relative merits. It does seem to swing back and forwards as to which is the highest performance. At the moment, I would say that for brute force, ASICS have it right now, but Intel have been demonstrating some really high throughputs with their DPDK framework, but I'm not sure how "real world" this is at the moment. There is no one "right" answer, and I'm aware of at least two vendors at the high-end (80Gbps+) that use ASICS, Merchant Silicon AND x86 to get the best of all three worlds..

Level 10

Real world? Such as? Everything I talk about here is currently being implement across many Firewall/NetSec vendors, it's just that I can't really get into the nuts and bolts of it in this context..If I write something else somewhere (as part of my dayjob) I'll share with the group!

Level 10

Thanks!

Level 11

I guess what I was saying was that I just wanted to see some examples and not just the overview, if you know what I mean. I hope you didn't take my wanting and wishing as unjust criticism!

Level 11

Thanks for sharing!     

Level 11

SSL, although necessary, has been the number 1 difficulty for me when it comes to DPI.  Great article.. keep them coming.

Level 14

Very good read!

Level 9

This post has very good information!

Level 11

I couldn't agree more.

Level 14

I have to admit that I'm struggling to recognize the use-case for an always-on DPI system that *wasn't* an appliance.  Don't get me wrong, I'm definitely a Solarwinds fanboy but I also know when some things are better left to dedicated hardware.  In my previous life I had a client that need to monitor 4 distinct networks with multiple ingress/egress points and one VERY large Internet pipe for their school district.  We recommended SourceFire.  (I can the hear the howls of "But Cisco owns them now!" already)  Why?  A scalable platform that allowed the customer to get some quick wins while not boxing themselves into a corner.  OK, the pricing was a little tough to swallow but when you're talking about security monitoring you are trying to buy the best insurance possible.

All that aside, I *really* love the idea of being able to enable DPI when and where it is needed without having to break open the piggy bank to do it.  I imagine enabling DPI along the same lines as using WireShark.  You'd never leave WireShark running all the time but you would use it as a targeted data collection tool.

That's my two cents.  Anyone care to ante up and raise me?

Level 21

I will ante up and raise you... but I am bluffing! 

I am not sure it fundamentally makes a difference if it's a system with an OS and an application or an appliance because at the end of the day it's ultimately the same thing accomplishing the same task.

I also am a self admitted SolarWinds fanboy and am really looking forward to how this is going to come together.  I also agree that DPI isn't normally something you run all of the time, it's generally reserved for troubleshooting scenarios.

Level 21

I take your point on how DPI is specifically designed to look for signatures that deviate from a known good.  SIEM is still probably much better situated to perform anomaly detection.

Level 14

The trickiest part about SIEM that I've found is knowing what "normal" is and knowing that "When A happens, and B happens within X amount of time, something bad is probably happening" when it's not something really obvious or that has happened before to show you the connection.

Level 12

this post just reaffirm my mantra that I have been pushing for some time now - generic out-of-the box IPS will not provide the security that may be expected.

instead targeted and customized packet inspection based on the target

great stuff and good read

keep more coming

Level 12

The problem with not leaving DPI (or WireShark) on all the time is that you have no historical data (trend) and you can fire them up while you have a problem but it's useless after the problem is gone.

Always on "surveillance" or "data gathering" is good for situations where someone tells you "Yeah, it does it now, but it did it yesterday and the day before too but I didn't call you...". You have no way of correlating if the 3 incidents are the same or related in any ways...

The NSA (probably) keeps everything on record so they can go back in time if they have suspicions. 

Level 13

Another important point is that SSL interception at the proxy/firewall is necessary for certain data loss prevention/monitoring platforms.

Level 21

That is certainly the balance; how much data to collect and store.  I personally don't think DPI data is something that is necessary to have on all of the time.  We already have ton's of other monitoring data from Orion and LEM; DPI can be busted out when troubleshooting is necessary.  I try to avoid being a digital hoarder and be deliberate about only collecting and storing as much data as it reasonably necessary understanding that there is always a cost with collecting and storing more data.  With that being said, I also realize that every environment and set of requirements is different so there are certainly cases where always on likely makes sense.

Level 11

likewise.

Level 10

Don't worry, I didn't take it as criticism at all, was just curious as to what you were interested in! The difficulty for me is that I work for a Firewall vendor, so whilst I can talk about the technology in a "how and why" sense, I can't really get too specific. That said, I can write about it as part of my day job, and if I know that here is interest out there for detail, I can address it in a later post elsewhere

Level 10

Agreed, especially on the "out of the box" part. My own mantra in terms of security is "you get out what you put in"; and IMO there is a lot of chasing of the newest "shiney shiney" in the business, but very little effort put into optimising the stuff you have.

Level 10

Yeah, if you hare going to have ANY hope of performing effective/meaningful DLP then yes, you need to be doing SSL inspection as a matter of course..

Level 13

I think it's worth making a distinction between DPI and full packet capture. Usually, DPI refers generically to any system that extracts metadata from network traffic at a higher layer than layer 4 -- in other words, you're getting application layer data that's more specific that TCP/UDP port number, ICMP message type, etc. Some examples:

  • HTTP headers and URIs
  • DNS queries
  • FTP commands
  • SSL certificate metadata extraction
  • TCP handshake time
  • TCP session time
  • HTTP transaction time

You generally get security incident results from DPI metadata by looking for suspicious patterns (DNS blacklist hits, uncommon FTP commands, self-signed certificates, weird user agent strings, etc).

With full packet capture, you're capturing all traffic or a subset to disk, all the time. Any serious incident detection program needs DPI metadata, but also needs to run full packet capture, all the time. If all you have is a signature-based IDS/IPS platform, you have no way to differentiate false and true positives the first time you see an alert. Your only hope in that case is to turn on full packet capture for that signature and wait for it to happen again -- which is really not a good way to approach it.

Level 14

I like seeing all the interaction here.

Level 12

A really interesting discussion on DPI jbiggley!


DPI has become a broad term now, anything from Wireshark to software or appliances which store metadata and/or full packets. Sometimes we need to step back and wonder why we really need this technology. I work for a DPI vendor and one thing I see more of is that DPI is used for is getting more visibility, deeper insight into

  • Bandwidth, all the detail you need to quickly ‘find that smoking gun’ to stop any finger pointing and save time, a level below NetFlow, a ‘NetFlow extender’
  • Security, malware, infected machines, machines sending SPAM, unusual user activity, list goes on…..
  • ‘Users and resources’, network usage, find out what users are doing on your network, find out how key resources including servers and the Internet are used.
  • Breakdown of traffic by application and user, not long list of port numbers.
  • Forensics, go back and understand exactly what happened, the detail, because DPI gives you that detail, the proof or evidence you need to say, it was not the network, it was Joe accessing a huge .pst file at 455 pm last Friday. Or it was this badly written application ‘look at all those queries from A to B’ Or it was John who moved that folder at 9 this morning….you can do all this via DPI, sometime people do not believe it until they see it.


All of this sounds great but the most important thing to figure out is where you need visibility. Once you have managed switches you can capture traffic anywhere. SPAN or mirror ports are gold when it comes to visibility but you need to be careful and not overuse them, a subject which I covered on my lovemytool.com blog


You can also install agents on servers but I would always favour the SPAN port method when it comes to continuous monitoring. Simply ‘grab the data off the wire’ without adding any load to already overburdened servers. SPAN ports are also handy when you don’t have flow options on your network devices.


Your choice of a DPI tool can come down to what level of visibility you require. What kind of information do you need to extract from the packets? Do you need timing information, application metadata, header information or all of the above? If you try and capture everything you may end up with too much detail and massive storage requirements. That’s the trouble with tools like Wireshark, as mentioned above, they are deployed when there is an issue and NOT left running continuously, way too much data.


But, it is possible to write software that makes the packets more readable, that uses DPI to recognise and extract the detail for the most popular applications, the metadata, the file names, the real domain names even if CDNs are being used, the URIs, the IP addresses and the user names as jswan mentions. However, this takes resources to capture, fingerprint, follow, reassemble, store and keep up. But, as also mentioned above a dedicated physical or virtual appliance is a very safe way to do this and very cost effective.


For me DPI is the next level of detail up from NetFlow, the visibility to ‘find that smoking gun’ immediately.  The critical information for every transaction but not every single packet.  Now you have the big picture but also the detail over longer periods of time, a level of detail you can now also use for forensics, capacity planning, reporting, proof for management that you need to make changes to your network.

Level 7

That makes the two of us Christopher.

Level 12

This is very good info.  Especially the encryption part.

Level 14

Thanks for the clarification.  I'm definitely geeked out by the prospect of DPI but I'm still not convinced that I would leave it running 24x7.  That said, if I had the computing resources and manpower resources to respond to a given trigger then I would definitely add it to my arsenal of data.

He who has the most data wins.  Isn't that what Mark Zuckerberg said or was that Sergey Brin?

Level 14

This is a killer explanation of DPI.  Thanks darragh.delaney for taking the time to lay all of that out.

If I opened the DPI rabbit hole I might never come back out!  It's like the first time I saw Netflow data --- oooooo sparklies!

Level 15

Thanks for posting this information.

About the Author
Glen Kemp is a professional services consultant for Fortinet Inc. in the U.K. His words are his own and do not necessarily reflect those of the company he works for. He designs and deploys network and application security tools, including access control, remote access, firewalls and other "keep the bad guys out" technologies. His blogs can be found at sslboy.net and at the Packet Pushers Podcast. Follow him on Twitter @ssl_boy.