Deep Packet Analysis - Fun with Flags!

In my last post I talked through some of the reasons why mastering tcpdump is useful. Building on our previous example in this post I’ll focus on using TCP flags for troubleshooting.

Even with our cleaned up filter, we can still see quite a lot of traffic that we don’t care about.  When troubleshooting connectivity issues, the first packet is the hardest; especially when you start involving firewalls. I’m sure you will recall, a TCP packet flagged with SYN is the first sent when client tries to establish a layer-7 connection to a remote host.  On a LAN, this is simple, but when the destination is on a different network, there is more to go wrong with the inevitable NAT and routing.

Troubleshooting approaches differ, but I prefer to jump on the target console and work backwards as traffic is usually only policed on ingress. We need to know whether our packet is reaching our target. This way we can look down the stack (at routing and firewalling) or up to application itself.

This is where a working knowledge of TCP flags and a copy of tcpdump is helpful. 

Signalling with Flags

Each TCP packet contains the source and destination port, the sequence  and acknowledgement numbers as well as a series of Flags (or control bits) which indicate one or more properties of the packet. Each flag is a single bit flipped on or off. The below diagram shows the fields in a TCP packet. The source and destination ports are 16 bit integers (and is where we get the maximum 65535 from), but if you look at offset 96, bits 8 through 15 are individual flag, SYN, ACK, RST, etc.

When a connection is setup between a client and server, the first three packets will have the SYN and ACK flags set. If you look at them in any packet analyser, it’ll look something like this:

Client -> Server (SYN bit set)

Server -> Client (SYN and ACK bits set)

Client -> Server (ACK bit set)

To make sure we are actually getting the traffic, we want to see the first packet in the connection handshake. To capture packets where only the SYN flag is set, we use the following command from the server console:

[~] # tcpdump -i eth0 -n 'tcp[tcpflags] & (tcp-syn) != 0 and port not 22’

The above tcp option filters on specific packet fields. In this case we are using tcpdump’s built-in shorthand to look at the bits associated with TCP SYN (the ‘Flags [S]’ in yellow). 

The other thing we are introducing is using ‘single’ quotes on the filter. This prevents the shell from trying to interpret anything within brackets.

In the below example we can see three packets sent ~500ms apart (also highlighted in yellow). You must consider that almost everything on the wire is filtered out, we are only hearing one side of the conversation.

Three packets with the same source and destination ports transmitted 500ms apart us a clue to what is happening. This is the typical behaviour of a client connection that received no response, assumed the packet was lost in transit and tried twice more.

What does the flag say?

Having taken this capture from the server; we know the outbound communication is working, so it unlikely that an intermediate firewall is causing a problem. My hunch is the server is not completing the connection handshake for some reason. A quick and dirty way is to check for TCP Reset packets; the host’s universal way of asking for a “do-over” and restarting the handshake. Hosts will respond with a TCP reset when there is no application listening; the lights are on; but no-one is home.

[~] # tcpdump -i eth0 -n 'tcp[tcpflags] & (tcp-rst) != 0 and port not 22'

I took the above captures a few minutes apart, but for every TCP SYN, there is a TCP RESET from the server. Whether the server is actually listening on any ports or interfaces on that target destination port (3333) is easily confirmed with:

[~] # netstat -na | grep 3333

If no results are returned, the service ain't running. When taking captures from a firewall, you should expect different behaviour. In 99% of cases if a packet doesn't a match policy; it will be dropped without an acknowledgement (ACK) or reset (RST) packet.

These are not the flags you are looking for

With the tcpflags option we can pipe in additional matches. For example, we can look for all traffic where the tcp-syn and tcp-ack flags are not set to 0.

tcpdump -i eth0 -n 'tcp[tcpflags] & (tcp-rst|tcp-ack) != 0 and port not 22'

However, everything with SYN or ACK set  doesn’t constitute much of a filter; you are going to be picking up a lot of traffic.

Rather than just filtering on source/destination ports, why do I care about TCP flags? I'm looking for behaviour rather than specific sources or destinations. If you are troubleshooting back-to-back LAN PCs, you wouldn't bother with all the Vexillology. However, with hosts either side of a firewall, you can’t take much for granted. When firewalls and dynamic routing is involved traffic may cross NAT zones or enter an unexpected interface.

It’s easy to switch to source/destination filtering once you've “found” the flows you are looking for; I try and avoid making assumptions when troubleshooting.

In my next post I’ll dig into the payload of two common protocols to see what we can learn about layer 7 using only tcpdump.