cancel
Showing results for 
Search instead for 
Did you mean: 

Deep Packet Analysis - What's in a Name (service)?

Level 10

In my last post I looked at how flags can pull useful information out of packet that we otherwise we might struggle to see.  This time, we’re going to use tcpdump to look into the actual applications.


The first application I'm going to look at is the humble Domain Name Service (DNS), the thing that needs to work flawlessly before any other application can get out of bed. Because DNS lookups are typically embedded in the OS IP stack, a packet capture is often the only way to get any visibility.


The NS in a Haystack


In my scenario, I suspect that something is amiss with DNS, but I'm not sure what. So to pick up just DNS traffic, we only need to capture with simple filter:


[~] # tcpdump -i eth0 -pn port 53


Capture.PNG


In the above example, even with minimal options selected, we can see some really useful information. The built-in decoder pulls out the transaction ID from the client (equivalent to a session ID), the query type (A Record) and the FQDN we are looking for. What is unusual in this example is that we can see not one, but two queries, about 5 seconds apart. Given that we are filtering on all port 53 traffic, we should have seen a reply. It would appear that my local DNS proxy (172.16.10.1) for some reason failed to respond. The client-side resolver timed out and tried the Google Public DNS. This may be a one-time event, but but it certainly bears monitoring. If the client configuration has an unresponsive or unreliable DNS server as first port of call, a the very least, this will manifest in a frustrating browsing experience.


Selection of the Fittest (and Fastest)


Selection of DNS servers is pretty important; I hadn't realised that my test Linux box was using Google as a secondary resolver. Whilst it is reliable; it’s actually four hops and a dozen milliseconds further away than my ISP service. When your broadband is as crappy as mine, every millisecond counts.


Anyway, as you can see, Google returns eight A records for google.co.uk; any of them should be fine.


Another thing to look for is what happens when we make an invalid query, or there is no valid response:


Capture2.PNG


In this case we get a NXDomain (non-existent domain) error. This case is an obvious typo on my part, but if we turn up the logging with the very verbose (vv) switch the response is still interesting:


[~] # tcpdump -i eth0 -pnvv port 53


Capture3.PNG


Highlighted above is the SOA (start of authority) record for the domain .ac.uk; this is far as the server was able to chase the referral before it got the NXDomain response. 


Edit - Contributor JSwan Pointed out a small mistake; I've fixed the below version.


Whilst a bunch of stuff is revealed with very verbose enabled; not all of it is useful. One thing to look for is the IP time to live (TTL); this shows how many hops the packet has made since leaving the source. If this number is low, it can be an indicator of routing problems or high latency (I did say it wasn't very useful!).


Furthermore, the DNS protocol-specific TTL can be seen highlighted in yellow, after the serial number in date format. The DNS TTL specifies how long the client (or referring server) should cache the record before checking again. For static services such as mail, TTLs can be 24 hours or more. However, for dynamic web services this can be as low as 1 second. TTLs that low are not a great idea; they generate HUGE amounts of DNS traffic which can snowball out of control. The moral is, make sure that the TTLs you are getting (or setting) are appropriate to your use-case. If you failover to your backup data centre, with a DNS TTL of a week, it will be a long time before all the caches will be flushed.


As JSwan points out in the comments, if you use the very very verbose switch (-vvv), for A records tcpdump will display the DNS TTL in hours, minutes and seconds:


Capture4.PNG


Apparently Google has very short TTL. Interestingly, tcpdump doesn't print the DNS TTL for NXDOMAIN result, although it is still visible in the capture.

Why is capturing in context important?


Imagine trying to troubleshoot connectivity for a network appliance. You have configured IP addressing, routing and DNS, but yet it cannot dial-home to it’s cloud service. Unless the vendor has been really thorough in documenting their error messages, a simple DNS fault can leave you stumped. Once again, tcpdump saves the day; even on non-TCP traffic. The built in protocol decoder gives vital clues as to what may be borking a simple communication problem.


In my next and final blog of this series, I’m going to look at another common protocol, HTTP.





27 Comments
Level 13

Minor correction: the TTL value you highlighted is the IP TTL, not the DNS TTL. In order to see the DNS TTL with tcpdump you need to use -vvv, and it shows up in brackets as a time value rather than a number of seconds like it does in some other tools:

sudo tcpdump -vvv -ni eth0 udp port 53 and host 8.8.8.8

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

19:46:26.512450 IP (tos 0x0, ttl 64, id 4920, offset 0, flags [none], proto UDP (17), length 60)

    a.b.c.d.55357 > 8.8.8.8.53: [bad udp cksum 0xa8de -> 0x6538!] 33458+ A? www.foobar.com. (32)

19:46:26.542042 IP (tos 0x0, ttl 47, id 28031, offset 0, flags [none], proto UDP (17), length 90)

    8.8.8.8.53 > a.b.c.d.55357: [udp sum ok] 33458 q: A? www.foobar.com. 2/0/0 www.foobar.com. [3h35m55s] CNAME foobar.com., foobar.com. [3h35m55s] A 69.89.31.56 (62)

Level 15

bookmarked again. excellent post sir.

Level 10

Ouch; I thought that TTL of 65 seconds was a bit odd; I knew the DNS TTL was there, but I guess I was looking in the wrong place. I'll fix this, with credit to you.

Level 11

Great info to ponder with....

Level 14

Sometimes, any port in a broadcast storm just doesn't work.  Filter, filter, filter.

Level 11

tcpdump is a great tool for troubleshooting but to much information can clutter the output.  Know what you want to capture before you capture the traffic and as network defender said filter, filter, filter.

Level 9

Thank you for the detailed post.

Level 17

Another extremely useful post.  Thank you for contributing this information.

Level 8

Interesting read. thank you

Level 10

once again a very good article, thanks and as before looking forward to what you are planning on looking at in the http packets

Level 21

While I agree this is a great post, I think it may be more well suited for a White Paper or How-To doc as it doesn't lend itself to much discussion.

Level 9

great info

Level 9

Excellent post.

Level 10

Mmm...Good point; this time I planned things a bit better up front, which meant that there has been less opportunity to go off the the tangents that are often so interesting..I'll try and do something about that in my next piece..

To that end, what you would like to see in the next post?

MVP
MVP

Excellent write up but then I am just echoing everybody else's comments.

Level 13

Thanks for the detailed instructions. This is extremely helpful, and I can see how the techniques described would carry over to other situations as well.

Level 9

Awesome post. Thanks for the insight.

Level 12

good read have to keep them one around.

Level 17

mind-blown.jpg

Level 20


Not a lot gets past Jay... js.

Level 13

It's merely a sad commentary on the amount of my life spent staring at pcaps...

Level 21

I guess I should have anticipated my critique would get put back on me to make a suggestion, well played sir! 

While I am not suggesting that these tried and true CLI tools will or should go away, I would be interested to see ideas and conversation around how the same functionality could be better built in to the more modern monitoring and security tools to provide the single pane of glass and context across the different data sets.  Using this could help drive better automation and help the Tier 1 guys with better triage capabilities... at least that would be the hope.

Level 11

great information

Level 10

thanks for a great post.  Food for thought (I I guess I'm already in Thanksgiving mode!) 

Level 12

Great information. Thanks for the post

Level 14

Very good post.  I will definitely be using this.

Level 15

Helpful to get a refresher on some of the old tools.  Thanks for the post.

About the Author
Glen Kemp is a professional services consultant for Fortinet Inc. in the U.K. His words are his own and do not necessarily reflect those of the company he works for. He designs and deploys network and application security tools, including access control, remote access, firewalls and other "keep the bad guys out" technologies. His blogs can be found at sslboy.net and at the Packet Pushers Podcast. Follow him on Twitter @ssl_boy.