cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Deep Packet Analysis - Practical applications with tcpdump

Level 10

Well hello there, returning like a bad penny, I am here to talk again about Deep Packet Analysis. In my last series of blogs I talked about the use-cases for Deep Packet Analysis but conspicuous by it’s absence was a lack of real world applications. This time I thought I would dust off my old-timey packet analysis skills and share some practical applications. I’ll focus on troubleshooting but no doubt we’ll wander into security and performance as well.


Whilst Solarwinds have some excellent tools for network performance management, there will be occasions where they won’t be available. For example, when troubleshooting remote systems without a full desktop or limited privileges. We’d like to think that those expensive “AcmeFoo Widget 5000” appliances use a custom built operating system. However, instead of an OS written in assembly by virgin albino programmers, it’s usually a headless Linux distribution with a fancy web GUI. As a result, the tools are pretty universal. Wireshark is the kind of tool that most administrator-types would have on their desktop. It has no end of fancy GUI knobs; click randomly long enough you are bound to find something noteworthy. However, when you don’t have access to a desktop or can’t export a complete dump, working with the available tools may be your only option. One of the most of basic, and powerful, is of course tcpdump. Available on most platforms, many vendors use it for native packet capture with a CLI or GUI wrapper.


How does Packet Capturing work?


A packet capture grabs packets from a network interface card (NIC) so they can be reviewed; either in real-time or dumped to a file. The analysis is usually performed after the event in something like Wireshark. By default, all traffic entering or leaving the NICs of your host will be captured. That sounds, useful, right? Well SSH or Telnet (you fool) onto a handy Linux box and run:


[~] # tcpdump

or if you are not root:

[~] # sudo tcpdump


And BWAAA. You get a visit from the packet inception chipmunk.

tcpdump 1.PNG


Filling your screen is a summary of all the traffic flowing into the host (press control+c to stop this, BTW).  This mostly consists of SSH traffic from your workstation, which contains the SSH traffic etc. In the days of multicore everything, this is not so much of a problem. On a feeble or marginal box an unfettered tcpdump can gobble cycles like some sort of Dutch personal transport recycling machine. To make what is flying past your screen readable, and stop your CPU from getting caned, a filter is needed.


So, to do a capture of everything except the ssh traffic, Use the following:


[~] # tcpdump not port 22

tcpdump 2.PNG

And BWAAA. Your second visit from the packet inception chipmunk (it’s a slightly smaller than the first, to simulate this just turn down the volume).


This time, you can see all the traffic heading into the NIC; and there is probably more than you thought. What’s not obvious is that tcpdump and other packet interception technologies operate in promiscuous mode. This is a feature of the NIC designed to assist in troubleshooting. A tonne of network traffic arriving at the NIC is not destined for the host. To save host CPU cycles this is silently ignored and is not passed up the stack. If your NIC was connected to a hub (or ethernet bridge) there would be a lot of ignoring. Even on a switched network there is a lot of broadcast noise from protocols such as ARP, NetBIOS, Bonjour, uPNP, etc. Promiscuous mode picks up everything on the wire sends it up the stack to be processed, or in our case, captured.


Troubleshooting with all this rubbish stuff floating around is difficult, but not impossible. If you intend analysing the traffic in Wireshark, filtering after the event is easy. However in our pretend scenario we can’t export files, but we can filter in-place with the --no-promiscuous-mode (or -p) option.


[~] # tcpdump -p not port 22


tcpdump 3.PNG


You’ll still see a lot of traffic from ARP and the like, but it should be much cleaner. Still firmly in the realm of Marmot-family filmic tropes, the process of packet capturing actually generates a lot of traffic. By default, tcpdump will try and reverse map IPs to hostnames in the traffic it collects. You could take the not port filter a bit further and exclude your DNS servers, or the DNS protocol entirely with:


[~] # tcpdump -p not port 22 and not port 53

or

[~] # tcpdump -p not port 22 and not host 8.8.8.8


But both of those are a bit clumsy, as whatever we are trying to fix may be DNS related. When tcpdump attempts this resolution a lot of secondary traffic can be generated. This may alarm a Firewall administrator; as the host may not normally generate outbound traffic.  Furthermore, DNS is sometimes used for data exfiltration. A host suddenly generating a lot of queries (caused by an “innocent” tcpdump) could cause unnecessary panic. DNS resolution can also play tricks; it’s showing you what the DNS server thinks the source or destination is; not what the packet headers actually say. The better option is just turn the darn thing off with the -n option:

[~] # tcpdump -pn not port 22

tcpdump 4.PNG


So there we have a nice clean real-time output of what is going on, but it’s still a bit vague. Don’t worry if you don’t understand what you are actually seeing here, we will come to that.


Again, the default behaviour is not that helpful. Unless you tell it otherwise, tcpdump will pick a NIC to capture on, usually defaulting to eth0. If you are dealing with a host with a single NIC, this is a good guess. However on a server or firewall, the traffic direction traffic and source NIC matter.


The final option I shall mention is to specify the interface on which to capture. The command:


[~] # ifconfig

eth0      Link encap:Ethernet  HWaddr 00:08:9B:BD:CC:9F 

          inet addr:172.16.10.220  Bcast:172.16.10.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:15590165 errors:0 dropped:104804 overruns:0 frame:0

          TX packets:14783138 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:532

          RX bytes:1908396023 (1.7 GiB)  TX bytes:906356383 (864.3 MiB)

          Interrupt:11


lo        Link encap:Local Loopback 

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:28772 errors:0 dropped:0 overruns:0 frame:0

          TX packets:28772 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:14490908 (13.8 MiB)  TX bytes:14490908 (13.8 MiB)



Will show you a list of configured interfaces. To tell tcpdump to only capture on one, just use the -i switch followed by the logical interface name.


[~] # tcpdump -i eth0 -pn not port 22


tcpdump 5.PNG


If you don’t know which interface the traffic is appearing, use the any switch to pick them all. This disables promiscuous mode at the same time, so you don’t need the -p option. However, I found this to be less than universal; its not not supported by every OS/build of tcpdump.


[~] # tcpdump -i any -n not port 22


So, there we have it. A cleanish packet capture which shows us what’s going on with the upper layer protocols.  In my next post I’ll dig a bit deeper with some filters and advanced switches for “on the spot” troubleshooting.




33 Comments
Level 12

good read keep it coming

Level 11

Good info, I use wirekshark on a daily basis and we have Riverbed ARX to capture traffic on the network.

I never used TCPDump, but I think I will try it out and compare it to what we have.

MVP
MVP

I like the focus lately on command line administration. tcpdump is a must-have and must-know for systems engineers who need a low-level view into networked communications.

Also, +1 for arcane humor. Well played.

Level 10

This is bookmarked for sure. Great read and very informative.

Level 9

good read

Level 15

solid post for sure! I use tcpdump more than wireshark; yet still rarely enough that I have to google commands that I continually forget

Bookmarked for reference. I look forward to next week!

MVP
MVP

Excellent !!!

Level 10

Thanks!

MVP
MVP

Very nice info here, thank you.

Level 10

Ta! There are plenty of really excellent tools out there these days, and to be honest I use Wireshark more than anything, but no matter how great your lighter is, sometimes all you have is a couple of twigs..

Level 10

Thanks! Reading that comment made my day! Off-topic a bit, but many are predicting the death of CLI, driven by improved API's, automation and SDN in general. But thinking about it, these old-school skills are going to be more relevant than ever - Can you just imagine the holy mother of bob screw-ups we're going to get with an automated, self-provisioning network? For the same reason that in the back of every pristine Mercedes dealer workshop, *somewhere* there will be a dirty great big hammer, CLI will is never going to go away.

Level 10

Yeah me to, I did have to double check a few things for this and the next article!, the other thing of course is that tcpdump is still being maintained; I discovered something from the MAN page wouldn't work because my geriatric SAN was using a *really* outdated build..

MVP
MVP

CLI 4 LYF

In case you haven't read it, check out this book. It's chicken soup for the CLI soul.

Level 17

Great overall information.  I think I fall into the same group as many where I've used WireShark more often than not, but this is solid information for an alternative that I'm looking forward to trying out.  Thanks glenkemp for this info.

Level 9

very helpful!

Level 9

Another valuable post. Thank you!

Level 21

Nice post.  What would be great is to have a tool such as NPM be able to perform that type of command and throw the output into a pop-up window much like SAM does with the real-time log viewer on Windows.

Level 17

This is a fantastic addition to our growing knowledge base. Thank you!

In almost every network class/tutorial/session, when the topic of packet capture is followed by the phrase "is outside the scope of this training".

With that said, this underscores the value that DPI in NPM 11 provides. Just like you can track bandwidth or CPU on all your devices using commandline options, having NPM makes the job so much easier. Nobody is saying you should forget the TOP command,

Ditto DPI. As your posts show, a competent ITPro can get packet capture and derive time-to-first-byte, TCP-three-way-handshake, and more. BUT having NPM with DPI set up makes that process soooo much easier.

Level 13

I've only rarely had to drop back to tcpdump, but the most memorable has been when working with Solaris admins troubleshooting name resolution on a remote box.  We were able to use tcpdump to show that the server wasn't even trying to query any DNS servers; in which case we can stop wasting time by looking at the firewall config.

Tcpdump isn't as pretty to look at as Wireshark, but it is more than sufficient when trying to confirm or refute basic connectivity.

Level 9

This is useful. Most of the time, I can pull a cap into Wireshark, which makes things easy, but this is useful for when that's not an option.

Level 13

Nice!


Fortunately I'm usually in situations where I can use Wireshark, but this is good information to have regardless.

Level 14

Very good article.  This is detailed and very useful information.

Level 14

wbrown, I have to disagree about the beauty of TCPdump.  With the terminal window background set to black and the text set to green, few things are more beautiful than watching packets scroll up the screen.

Level 13

That's because you're just paying attention to the woman in the red dress.

Level 14

Very well written article!  I use TCPdump extensively as a firewall admin.  I will pull open a pair of shells, one for each interface on the firewall that requires my attention.  I usually know the IP's of interest and the port as well.  tcpdump -npi 1-0 host x.x.x.x and host x.x.x.x and port xx.  This will show the conversation of interest on the interface of interest (1-0) and exclude everything else.  The firewall has a gui that will show the audit files and is of some help, but I would be relying on the gui to interpret the traffic for me.  I prefer to see the raw traffic and do my own interpretation.  Of course, the shell sessions must have a black background and green text.  I find that it is easier on the eyes when doing packet analysis for an extended time.

SANS used to teach a half day packet analysis class that was excellent.  I have an awesome packet analysis reference guide that I refer to on a regular basis.  If you would like a copy let me know.

Level 8

Great article!

Level 10

Good info, thanks

Level 11


A pleasure to read and learn.

Level 12

tcpdump is often overlooked just because it's usually misused. "Too much data" and "I can never find anything useful" are the comments we hear the most often. I've been passing for an alien more often then not with my command prompt/ssh/terminal window always open, but again, I'm usually the one who finds the solution the first.

Thanks a lot glenkemp, very nice article !

Level 17

A Good post is one that can stand the test of time, and landing in by bookmarks it certainly does that. Often my bookmarks get filled with pages chocked full of tips, tools and a certain type of insight or outlooks that networks seem to require. This will be one of those sites.

Level 10

very nice and easy to ready article, tcpdump is definitely a great resource tool to have knowledge about and one that many people sadly look over

Level 14

We use a Opnet (Riverbed now) AppResponse Xpert (ARX) platform do capture and slice and dice the packets.  It actually uses wireshark to do some of the analysis, but it's great seeing the method you have shown above.  Thanks for the great post.

Level 15

Good post.  Could definitely use more of this type of post.  Thanks for sharing.

About the Author
Glen Kemp is a professional services consultant for Fortinet Inc. in the U.K. His words are his own and do not necessarily reflect those of the company he works for. He designs and deploys network and application security tools, including access control, remote access, firewalls and other "keep the bad guys out" technologies. His blogs can be found at sslboy.net and at the Packet Pushers Podcast. Follow him on Twitter @ssl_boy.