No, this isn't a post about hunting or the last time I played Battlefield Bad Company and no I'm not the next guest on Bear Grylls' hit TV show Man vs. Wild (if you're reading this Bear I'm still waiting on that invitation in the mail)... What I'm talking about is what we as IT managers, system administrators, and network engineers do every day - we search and destroy. We hunt, track, and eventually locate problems in our IT environments and then we take them out.

Some of the most troublesome things to locate are devices and users' machines on the network. Typically, when you identify the problem you start out with an IP address. You're looking through some NetFlow data and you see that the user or device at 192.168.54.12 is using up 90% of your internet bandwidth downloading videos from iTunes. You really need to know who that user is (or whose machine it is) before you take action else you might be writing an access list to block your own CEO from accessing the internet and that's never a bright idea (sorry about that Kevin, I really didn't know it was you). So, what do you do next?

Well, it really depends on what tools you have at your disposal. You should be able to trace that IP to a particular subnet fairly easily. Once you know which router that IP is using as a first hope (layer 3 wise I mean) you can look through the router's ARP cache to match the IP address up to a MAC address, assuming that the user is still sending internet traffic and the ARP table entry hasn't timed out. If you don't see a valid ARP table entry you can try pinging that IP from the router to populate the cache. So long as the machine is still on and using that IP address that should get you a valid MAC address.

Now that you've got a MAC address to work with you can login to the switch and start looking thru bridge tables (CAM tables on Cisco switches) to see which port that machine is connected to and hopefully after that you'll be able to map it to an office or cubicle location.

That's a lot of work and it's sort of a best case/easiest resolution scenario so let's complicate things a bit. Last Thursday someone from your company hacked into one of your customers networks and starting downloading pornographic material to one of their web servers. Hey, free bandwidth and disk space, right? Well, thanks to that ******** your CIO is breathing down your neck. You're able to tell from looking at some of your management tools that the IP address was 192.168.55.17. However, that IP address is in a DHCP pool with 2 day lease times and that particular address block is used by a bunch of temporary employees that all bring in their own personal laptops and connect to your corporate network. Say goodbye to your lunch break my friend...

Problems like these happen all the time and in many cases you're in a real hurry to solve them. One time, many years ago before SolarWinds, I had a user that had hacked into one of my mail servers and was using it to download gigabytes of images that he'd found online. Not only was this causing problems for the mail server but it was choking our internet bandwidth and this was happening while we were attempting to finish some finance updates that were time sensitive. We literally had only a few minutes to solve this problem or risk missing the market close and probably our jobs.

If you've had experiences like this or if you have tips on solving these types of problems post a comment and share with the rest of us. In Part 2 next week I'll discuss some of the solutions available to help solve this problem and I'll post some "sneak peak" footage of a new product from SolarWinds.


Flame on...
Josh
Follow me on Twitter