Deep Packet Analysis - Practical applications with tcpdump

Well hello there, returning like a bad penny, I am here to talk again about Deep Packet Analysis. In my last series of blogs I talked about the use-cases for Deep Packet Analysis but conspicuous by it’s absence was a lack of real world applications. This time I thought I would dust off my old-timey packet analysis skills and share some practical applications. I’ll focus on troubleshooting but no doubt we’ll wander into security and performance as well.

Whilst Solarwinds have some excellent tools for network performance management, there will be occasions where they won’t be available. For example, when troubleshooting remote systems without a full desktop or limited privileges. We’d like to think that those expensive “AcmeFoo Widget 5000” appliances use a custom built operating system. However, instead of an OS written in assembly by virgin albino programmers, it’s usually a headless Linux distribution with a fancy web GUI. As a result, the tools are pretty universal. Wireshark is the kind of tool that most administrator-types would have on their desktop. It has no end of fancy GUI knobs; click randomly long enough you are bound to find something noteworthy. However, when you don’t have access to a desktop or can’t export a complete dump, working with the available tools may be your only option. One of the most of basic, and powerful, is of course tcpdump. Available on most platforms, many vendors use it for native packet capture with a CLI or GUI wrapper.

How does Packet Capturing work?

A packet capture grabs packets from a network interface card (NIC) so they can be reviewed; either in real-time or dumped to a file. The analysis is usually performed after the event in something like Wireshark. By default, all traffic entering or leaving the NICs of your host will be captured. That sounds, useful, right? Well SSH or Telnet (you fool) onto a handy Linux box and run:

[~] # tcpdump

or if you are not root:

[~] # sudo tcpdump

And BWAAA. You get a visit from the packet inception chipmunk.

Filling your screen is a summary of all the traffic flowing into the host (press control+c to stop this, BTW).  This mostly consists of SSH traffic from your workstation, which contains the SSH traffic etc. In the days of multicore everything, this is not so much of a problem. On a feeble or marginal box an unfettered tcpdump can gobble cycles like some sort of Dutch personal transport recycling machine. To make what is flying past your screen readable, and stop your CPU from getting caned, a filter is needed.

So, to do a capture of everything except the ssh traffic, Use the following:

[~] # tcpdump not port 22

And BWAAA. Your second visit from the packet inception chipmunk (it’s a slightly smaller than the first, to simulate this just turn down the volume).

This time, you can see all the traffic heading into the NIC; and there is probably more than you thought. What’s not obvious is that tcpdump and other packet interception technologies operate in promiscuous mode. This is a feature of the NIC designed to assist in troubleshooting. A tonne of network traffic arriving at the NIC is not destined for the host. To save host CPU cycles this is silently ignored and is not passed up the stack. If your NIC was connected to a hub (or ethernet bridge) there would be a lot of ignoring. Even on a switched network there is a lot of broadcast noise from protocols such as ARP, NetBIOS, Bonjour, uPNP, etc. Promiscuous mode picks up everything on the wire sends it up the stack to be processed, or in our case, captured.

Troubleshooting with all this rubbish stuff floating around is difficult, but not impossible. If you intend analysing the traffic in Wireshark, filtering after the event is easy. However in our pretend scenario we can’t export files, but we can filter in-place with the --no-promiscuous-mode (or -p) option.

[~] # tcpdump -p not port 22

You’ll still see a lot of traffic from ARP and the like, but it should be much cleaner. Still firmly in the realm of Marmot-family filmic tropes, the process of packet capturing actually generates a lot of traffic. By default, tcpdump will try and reverse map IPs to hostnames in the traffic it collects. You could take the not port filter a bit further and exclude your DNS servers, or the DNS protocol entirely with:

[~] # tcpdump -p not port 22 and not port 53

or

[~] # tcpdump -p not port 22 and not host 8.8.8.8

But both of those are a bit clumsy, as whatever we are trying to fix may be DNS related. When tcpdump attempts this resolution a lot of secondary traffic can be generated. This may alarm a Firewall administrator; as the host may not normally generate outbound traffic.  Furthermore, DNS is sometimes used for data exfiltration. A host suddenly generating a lot of queries (caused by an “innocent” tcpdump) could cause unnecessary panic. DNS resolution can also play tricks; it’s showing you what the DNS server thinks the source or destination is; not what the packet headers actually say. The better option is just turn the darn thing off with the -n option:

[~] # tcpdump -pn not port 22

So there we have a nice clean real-time output of what is going on, but it’s still a bit vague. Don’t worry if you don’t understand what you are actually seeing here, we will come to that.

Again, the default behaviour is not that helpful. Unless you tell it otherwise, tcpdump will pick a NIC to capture on, usually defaulting to eth0. If you are dealing with a host with a single NIC, this is a good guess. However on a server or firewall, the traffic direction traffic and source NIC matter.

The final option I shall mention is to specify the interface on which to capture. The command:

[~] # ifconfig

eth0      Link encap:Ethernet  HWaddr 00:08:9B:BD:CC:9F 

          inet addr:172.16.10.220  Bcast:172.16.10.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:15590165 errors:0 dropped:104804 overruns:0 frame:0

          TX packets:14783138 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:532

          RX bytes:1908396023 (1.7 GiB)  TX bytes:906356383 (864.3 MiB)

          Interrupt:11

lo        Link encap:Local Loopback 

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:28772 errors:0 dropped:0 overruns:0 frame:0

          TX packets:28772 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:14490908 (13.8 MiB)  TX bytes:14490908 (13.8 MiB)

Will show you a list of configured interfaces. To tell tcpdump to only capture on one, just use the -i switch followed by the logical interface name.

[~] # tcpdump -i eth0 -pn not port 22

If you don’t know which interface the traffic is appearing, use the any switch to pick them all. This disables promiscuous mode at the same time, so you don’t need the -p option. However, I found this to be less than universal; its not not supported by every OS/build of tcpdump.

[~] # tcpdump -i any -n not port 22

So, there we have it. A cleanish packet capture which shows us what’s going on with the upper layer protocols.  In my next post I’ll dig a bit deeper with some filters and advanced switches for “on the spot” troubleshooting.