Skip navigation

Geek Speak

4 Posts authored by: kmillerusaf

There is much talk in the IT profession about automation. “Automate all the things” is written in some shape or fashion across a variety of blogs and social media platforms. I even briefly mentioned it in my last Geek Speak post about configuration management. You have read that already... Right?

 

5b3761867c14de76ab959f4b9ece9a3b51654222b93c92e7e02c4e80fad9da21.jpg

 

I get the movement. Why do everything manually, wasting your time on tedious, trivial tasks when you could be working on the newest design for your data center or something better? And even though I could probably consider myself a new age networking professional, there’s still one task I enjoy doing the old-fashioned way: network discovery.

 

Call me crazy, but the task of learning a network for the first time in my opinion is best done manually. There are so many nuances that could be lost if this process is done automatically. Dissecting a network, one device at a time, port by port truly allows the ability to intimately understand the complexities of that network. Here are some tips and tricks that I have learned along the way and also seen other networking professionals speak of when discovering a network for the very first time:

 

  • Start from the Core switch and work your way out (if you don’t know where the Core switch is, start with your default gateway and branch out)
  • Use information like CDP/LLDP, ARP, MAC-addresses, and routing tables to help you navigate
  • NEVER completely trust switch port descriptions or network diagrams. They are almost always not kept up with or updated regularly.
  • Draw out the network as you go using pencil and paper. You will continuously edit this diagram so using pen will hamper you and trying to input this into a program like Microsoft Visio while continuously making changes will make you scream.


What about you all? Do you prefer automated network discovery or would you rather do it manually? Have any tips for either method? I look forward to hearing from you all.

I’ll be honest, when I initially saw the words configuration management, I only thought of managing device configurations. You know, things like keeping backup copies of configurations in case a device bit the bucket. However, the longer I’ve been in the IT field, the more I’ve learned how short-sighted I was in relation to what configuration management truly meant. Hopefully, by the end of this post, you will either nod and agree or thank me for opening your eyes to an aspect of IT that is typically misunderstood or severely neglected.

 

There are several components of configuration management that you, as an IT professional should be aware of:

 

  • Device hardware and software inventory
  • Software management
  • Configuration backup, viewing, archiving, and comparison
  • Detection and alerting of changes to configuration, hardware, or software
  • Configuration change management

 

Let’s briefly go over some of these and why they are so integral to maintaining a healthy network.

 

Most (hopefully all) IT teams keep an inventory of hardware and software that they support. This is imperative for things like service contract renewals and support calls. But, how you keep track of this information usually calls for question. Are you manually keeping track of this information using Excel spreadsheets or something similar? I would agree that it works, but in a world so hellbent on automation, why risk human error? What if you forget to add a device and it goes unnoticed? Wouldn’t it be easier to have software that automatically performs an inventory of all your devices?

 

One of my favorite components of configuration management is configuration backup and the ability to view those backups as well as compare them to previous backups. If your Core switch were to fail today, right now, are you prepared to replace it? I’m not talking about calling your vendor’s support to have them ship out a replacement. I’m talking about rebuilding that new shiny piece of hardware to its predecessor’s last working state. If you have backups, that process is made easy. Grab the latest backup and slap it on the new device when it arrives. This will drastically cut down the recovery time in a failure scenario. Need to know what’s changed between the current configuration and 6 months ago for audit purposes? Having those backups and a mechanism for comparing them goes a long way.

 

There are a number of ways to know when an intruder’s been in your network. One of those methods is through the detection and alerting of changes made to your devices. If you don’t have something in place that can detect these changes in real-time, you’ll be in the dark in more ways than one. How about if a co-worker made an “innocent” change before going on vacation that starts to rear its ugly head? Being able to easily generate real-time alerts or reports will help pinpoint the changes and get your system purring like a kitten once again.

 

In conclusion, configuration management is not just about keeping backups of your devices on hand. It involves keeping inventories of those devices as well as being able to view, archive, and compare their configurations. It also includes being able to easily detect and alert on changes made to your devices for events like catching network intruders. Are you practicing good configuration management techniques?

IT professionals are admittedly a prideful bunch. It comes with the territory when you have to constantly defend yourself, your decisions, and your infrastructure against people who don’t truly understand what you do. This is especially true for network administrators. “It’s always the network.” Ever heard that one before? Heck, there’s even a blog out there with that expression created by someone I respect, Colby Glass. My point is, as IT professionals, we have to be prepared at a moment’s notice to provide evidence that an issue is not related to the devices we manage. That's why it's imperative that we must know our network very well inside and out.

 

With that being said, It should be no surprise to you that when I started my career in networking in 2010, I thought NMS platforms were pretty amazing. Pop some IP addresses in and you’re set.¹ The NMS goes about its duty, monitoring the kingdom and alerting you when things go awry. I could even log in and verify it for myself by looking if I wanted to be certain. I could even dig in at the interface level and give you traffic statistics like discards and errors, utilization, etc. I had instant credibility at my finger tips. I could prove the network was in great shape at a moment's notice.  Want to know if that interface to your server was congested yesterday evening at 7pm? It sure wasn't and I have the proof! Can’t get much better than that, right?

 

Until…

 

I saw netflow for the first time. Netflow has a way of really opening your eyes. “How did I ever think I knew my network so well in the past?”, I thought. I had no visibility into the traffic patterns flowing through my network. Sure, I could fire up a packet capture pretty easily, but that approach is reactive and time-consuming depending on your setup. What if that interface really WAS congested yesterday evening at 7pm? I have no data to reference because I wasn't running a packet capture at that exact time or for that particular traffic flow. It’s helpful to tell someone that the interface was congested, but how about taking it a step further with what was congesting it? What misbehaving application caused that link to be 90% utilized when traffic should have been relatively light at that time of the day? The important thing to realize is that I’m not just an advocate for netflow, I’m also a user!² Here’s a quick recap of an instance where netflow saved my team and I.

 

I recently encountered a situation where having net flow data was instrumental. One day at work, we received multiple calls, e-mails, and tickets about slow networks at our remote offices. They seemed to be related, but we weren't sure at first. The slowness complaints were sporadic in nature which made us scratch our heads even more. After looking at our instance of NPM, we definitely saw high interface utilization at some, but not all of our remote sites. We couldn't think of any application or traffic pattern that would cause this. Was our network under attack? We thought it might be prudent to involve the security team, in case it really was an attack, but before we sounded the alarm, we decided to check out our netflow data first. What we saw next really baffled us.

 

Large amounts of traffic (think GBs/hour) was coming from our Symantec Endpoint Protection (SEP) servers to clients at the remote offices over TCP port 8014. For those of you who have worked with Symantec before, you probably already know that this is the port that the SEP manager uses to manage its clients (e.g. virus definition updates). At some point, communication between the manager and most of its clients (especially in remote offices) had failed and the virus definitions on the clients became outdated. After a period of time, the clients would no longer request the incremental definition update; they wanted the whole enchilada. That’s okay if it’s a few clients and the download process ends in success the first time. This wasn't the case in our situation. There were hundreds of clients all trying to download this 400+MB file from one server over relatively small WAN links (avg. 10Mb/s). The result of this was constantly failing downloads which triggered the process to start over again ad infinitum. As a quick workaround, we decided to QoS the traffic based on the port number until the issue with the clients was resolved. With this information at our disposal, we brought it to the security team to show them that their A/V system was not healthy. Armed with the information we gave them, they were quickly able to identify several issues with the SEP manager and its clients which helped them eventually resolve several issues including standing up a redundant SEP manager. Without net flow data, we would have had to set up SPAN ports on our switches and wait  for a period of time before analyzing packet captures to determine what caused the congestion. By having netflow, we were instantly able to capitalize on it by viewing specific times in the past to determine what was traversing our network when our users were complaining.

 

That’s just one problem netflow has solved for us. What if that port was TCP/6667 and it was coming from your CFO’s computer? Do you really think your CFO is on #packetpushers (irc.freenode.net) trying to learn more about networking? No, it’s more likely a command and control botnet obtaining its next instructions on how to make your life worse. From a security perspective, netflow is just one more tool to add in the never-ending fight against malware. So what are you waiting for? Get with the flow… with netflow!

 

1. Of course it's never quite that easy. You'll have to configure SNMP on all of your devices that you want to manage and/or monitor.

2. Hair Club for Men marketing

I’ve worked with a few different network management systems in my career, some commercial and some open source. Based on my experience with each one of them, I’ve developed certain qualities that I look for when deciding on which product to recommend or use.

 

One common theme that always seems to come up when comparing experiences with my peers is how easy it is to implement and operate product $A vs product $B?

 

In my opinion, implementation and operation of a product are critical when accessing any product, not just a NMS. If it’s not easy to implement, how are you ever going to get it off the ground? Will training be necessary just to install it? IF you ever do get it off the ground and running, will it take a small army to keep it going? Will you have to dedicate time and resources each day just to cultivate that product? How can you trust a product with the management and monitoring of your network environment, if it crashes all the time or if you have to be a Jedi to unlock all of its mystical bells and whistles?

 

With that being said, what do you look for in a network management system? Easy to install? Intuitive interface? All the features you could wish for? Is cost the ultimate factor? I’d love to hear what you all think.

 

Regards,

Keith

Filter Blog

By date: By tag: