cancel
Showing results for 
Search instead for 
Did you mean: 

The Two Faces of NetFlow

Level 12

Enabling NetFlow will give you some insight on what your network actually carries

-- Nicolas Fischbach in Black Hat conference

Even though we discuss NetFlow in this article, the content also applies to other flow technologies: J-Flow, sFlow, NetStream, etc.

In the discussion of my first June Ambassador blog post The Cost of InfoSec Stewardshipjswanprovided a great idea of reducing information $ecurity costs: implementing solutions that can be used for multiple purposes. He stated, for example, that NetFlow could be used by multiple departments in an organization like Operations, Security, Networking, and Help Desk.

My organization is mainly a Cisco shop, so we implement NetFlow. Since I split my working hours in Network Security and in Data Center / Campus Networking, I have opportunities to use NetFlow as an information security tool and a network performance tool. We, as many organizations, were introduced NetFlow analyzer by different vendors as a security tool. NetFlow analyzer vendors know that many organizations lack in knowledge of what's going on in their network. The vendors also know that by showing the executives the unexpected Top Talkers in the network after one or two days of the POC, the executives will be convinced to pull out the checkbook.

The NetFlow solution for security doesn't come cheap. The cost of the NetFlow analyzer is one thing. You need FULL NetFlow, rather than SAMPLED NetFlow, for network forensics. If you have a scale-out network, you'll need multiple flow collectors and in turn you'll need more storage. In the end, it is a good idea to present to the CIO that this solution is multi-purpose.

Do you want to hear a true story of the "alternative" usage of NetFlow? A Windows server admin accidentally clicked "Go" in "Default Server" of the Rapid Deployment System. Immediately hundreds of servers were… "defaulted" and started PXE boot. Countless alerts showed up in the NOC monitoring system. Within five minutes, the IT managers of different departments stormed in the poor network manager's office and asked what's wrong the network (pretty common, I guess). Executives commanded to reboot this switch and that router. After the pale-face Windows admin confessed his mistake to the people, everyone didn't know where to start to identify all damaged servers in the next 45 minutes.

The NetFlow guy in another office was notified about the incident. He calmly ran a NetFlow report for all PXE boot traffic for the period of the incident. That report saved many lives that day.

Does your organization implement NetFlow or any other flow technology for information security?

Is that technology also used for something other than security?

Do you have any story to share?

I hope your story is not that scary.

59 Comments
Level 13

If you understand your network topology, NetFlow is probably the single most useful tool you can have after basic performance monitoring/alerting. The biggest problems people run into at the beginning usually have to do with not really understanding their own network topology -- this is common in cases where the network was built by consultants and then left to the operators without much training or documentation. Because NetFlow is fundamentally oriented towards interfaces rather than nodes, it's easy to get confused about the direction of traffic flow or the routed path taken by a flow. Most flow analyzers have a way to summarize interface data into node data, but often this just makes things even more confusing. When people first get a flow analyzer, I encourage them to look at it with a network map in hand.

Also, it's handy to keep in mind that you can use NetFlow features at the CLI without a collector or analyzer. In the Cisco world, the "show ip flow top-talkers" and "show ip cache flow" commands are a great source of data if you have NetFlow configured on the device, even if it's not exporting.

Level 9

That last point was something I didn't even realize.  Thanks for smacking that one into my brain.

Level 12

thanks for the post mfmahler

Level 11

We don't use NetFlow we use F5.

Level 12

Hari Pala, would you spend some time here and elaborate? Thanks!

Level 12

Thank you, esther.

Level 12

jswan Good info and well said, as always! You pointed out the key of NetFlow which is interface-oriented.

If the network is done by consultants, detailed documentation should be in SOW. If no detailed documentation is delivered, no payment.

Level 11

We currently are collecting Netflow data but at this time its not used in our IT security departments.  I would say that other than finding out who is the big bandwidth hog at the moment, most of our Netflow at this point is just getting playground use.  We are picking and poking at it trying to determine what good it will be for us in our environment.  As our network gets more and more complex, I believe that the hope is netflow can show us what traffic exists where. 

Jim

Level 10

We use NetFlow currently against an F5, and some other devices. It was wonderful when one of our clients was using BitTorrent to download a popular television show. We were alerted of the incident from our ISP. We were very easily able to track that user down showing exactly what time they used BitTorrent, and from which machine.

We then setup alerts for this traffic. Now we get alerted to when someone used BitTorrent. We let our clients know ahead of time, so that if they are using it for something they should not be, we can put a stop to it very quickly.

We do. We implement sFlow internally and Netflow from managed routers.

It's used for security, bandwidth planning and management, and other non-technological purposes such as HR.

I've probably already shared this small anecdote, but here goes:

it was my first week in the shop. I noticed on my first day we had zero monitoring in place. I also overheard my boss speaking with someone about mysterious business-hours latency between two of our MPLS sites.

Two days later, I had a trial of NPM in place and sFlow configured on core switches at the two locations. I isolated, identified, and solved an issue plaguing our company for months within fifteen minutes.

Nothing special skillwise - anyone could do it - but I do feel it illustrates the value of the technology!

Level 12

rharland2012 Thank you for sharing your story! It's awesome to resolve a long time issue in a matter of minutes!

Yes, it does require skills and understanding of the technology, plus having the right tool(s). As jswan stated, it also needs knowledge of the network topology. You are good.

Level 12

Corey, that is cool! You are talking about being proactive here! And people can't lie about what's on the wire (network).

Level 12

Jim, thank you for stopping by here!

You know what? You can sell the NetFlow solution to the security departments and the CIO, and get more budget to upgrade/expand the NetFlow system. Then NetFlow integrates to AD. The NetFlow feeds to the SIEM. Wow!

Level 12

you are most welcome mfmahler

Level 12

Thank you for sharing rharland2012

MVP
MVP

We have netflow operating throughout our network. Not only for security reasons but also to identify the traffic going across our circuits. Netflow is a great thing to implement and we get a lot of insight into exactly what is transverseing our network. We use that information to implement ACL's to block traffic that is non-work related in nature. You would be surprised at the number of non-work related sites that users are going to during working hours.

Level 9

To me this would be a double edge sword.   This is based on the question I would ask - do the security teams, HR and others need this data?    The answer depends on your / their environment.    If your company is a bank or retail, medical ect then I would think forensically this information would be invaluable in tracking down breaches.   However, how much is too much?    How many of us get alerts we just ignore?   We know that there is really nothing wrong and just do not have time to fix the problem so we ignore it and move on.     I think that data such as this would be the same for those people.   You need buy in from more than just 1 or 2 people who would / could use it.     Then those who do use it need to know what they are looking at.

Level 12

Food for thought!!!!! Thanks bruce.jarrett

At least in our environment, delivery of this data to those departments is not the default. Occasionally, we are called upon to leverage xFlow data - and our contextual knowledge of what it means - to bring information to the table in various situations. It would irresponsible and lazy to dump xFlow reports on people, and I didn't intend to imply that was the case.

To respond to some of your other questions:

How much is too much?

I'm not sure - I definitely know what 'not enough' is, though.

How many of us get alerts we just ignore?

Most of us, I'm sure.

Level 12

we dont use netflow our network team is all about the cisco prime and what they have to offer.

Level 12

Yes, dwoj, it's there. Check it out on a NetFlow enabled switch/router.

Level 12

Aaron Denning We use Cisco Prime, too. Both tools provide different functions. Show your Network Team this blog post. They may have a second thought about using NetFlow.

Level 9

I work most of my time as a consultant with customers and it's incredible the percentage of networks poorly or not at all documented (at least here in Mexico) NetFlow can definitely serve as a powerful tool to start "knowing" this networks, so the troubleshooting of issues can be a lot faster and simpler. The problem here is that most companies don't value correctly the need for network management and visibility tools like NTA. It's our job as consultants to show them what they are missing and the high costs of not having the right tools to solve the network problems. Great post jswan!

Level 12

We have but they don't take well to change... if they set there mind that this is it then thats it.. i just have to try to figure the rest out as i go and just hope i don't screw anything up.

It's hard to screw anything up with xFlow.....it's just like picking a book off a shelf and reading it. The bookcase shouldn't fall down!

Level 9

What if you can read the language but cannot make sense of they syntax?

That is where I have problems giving people more information than they

know what to do with / look at. Then you get into problems. Using

NetFlow for Forensics is great but for proactive monitoring for security

and such that is very difficult to judge because remember is it not real

time it is a snap shot in time.

On Tue, Jun 17, 2014 at 11:54 AM, rharland2012 <

As I said, we are occasionally called upon to leverage xFlow data - and our contextual knowledge of what it means - to bring information to the table.

These are usually situations where we're examining recent historical events.

I don't think anyone in this conversation thinks xFlow is realtime - I certainly don't, and didn't read anything else on here implying that it is.

Level 13

I'm not sure what you mean by this. If you are doing unsampled NetFlow with a 60-second active flow export timer, you see traffic events in near real-time. The flow records themselves are accurately timestamped, so you have sub-minute granularity if you need it. Not all analyzers give you this degree of granularity (NTA, for example, doesn't), but the data is still there.

Maybe you were talking about the case where you need layer 7 traffic visibility. In that case, you have little choice but to do full packet capture, but there are now some IPFIX exporters that support limited Layer 7 traffic export capabilities:

  • Cisco IOS in 15.2T and later supports export of HTTP host headers and URIs, as well as several types of application transaction timers.
  • Palo Alto firewalls support HTTP host header export and possibly some other features.
  • YAF (an open source IPFIX probe from CMU-CERT) supports HTTP and DNS exports.
  • nProbe supports HTTP, DNS, and SQL query exports in IPFIX.

There are not a lot of collectors out there that support receiving this type of data, however.

Level 20

I'm currently using netflow on 6513 most recently to determine the volume and geographic location of multiple users of a large enterprise application.  We knew how many user accounts there were in different places.... what we didn't know for sure was how much they each were using and exactly where the power users were.  Netflow enables seeing more of the picture.

Level 20

We use OpsNet for this... and man let me tell you... it's a LOT of data o.O to have anytime deep packet inspection capability after the fact anytime anywhere.

Level 17

Awesome mfmahler !! Netflow is spotty at the moment and I don't operate it specifically.

Level 12

Thank you, cahunt! I'm curious of your thought about NetFlow being spotty.

Level 12

tell that to my network guys they are very stuck in there ways.

Level 12

Totally agree with you, Kurt H.

For web related traffic, we implement web proxy with WCCP, plus DLP with ICAP. The web traffic's visibility and control form there.

Level 12

bruce.jarrett My team owns, implemented, and maintains NetFlow and its analyzer. We started NetFlow as a security tool, as my team is network security. The data is always there. Yesterday an executive wanted a report for our WAN links utilization. The report is right at our fingertips.

As ecklerwr1 mentioned somewhere in this blog post thread, there is a lot, lot of data.

Level 12

ecklerwr1 That's an awesome NetFlow use case!

Level 17

it isn't great. If something major happens we will have part of the puzzle to put together. And troubleshooting using the netflow stay's between B/E, Distribution & Core mainly.

My "only hope" is info-sec's new toy will fill in the gaps that are not covered by Netflow. They are no Obi-Wan!

Level 12

Aaron Denning It may be time to sell NetFlow to your infosec guys instead.

Level 12

jswan Thank you for sharing the IPFIX information and its cross-platform features.

Level 12

will have to work that angle thanks

Level 11

good post!

I hear you loud and clear on that....but as a network guy myself, I know that with a little planning, I was able to roll out xFlow safely with no user impact - even on WAN pipes that aren't that big.

Just my experience, though - I'm sure there are some folks on here who may have had some undesirable results.

Level 21

I can certainly see the value in NetFlow.  We have NetFlow tools in place and have used them during active troubleshooting scenarios.  With that being said I find myself split on what to think of it.  The data it provides when needed is always incredibly valuable; however, we have not found a significant number of situations where it was really needed which of course needs to be offset by the cost of having it.  There is always the option to only have NetFlow setup in specific places on your network or only set it up when you need a deeper dive to keep costs down; however, often times the value is lost if you don't have both the before and after data or the data from the right section of the network.

I personally have found having network devices log data to a SIEM solution to be a bit better balance.  While I admittedly don't have nearly the level of data NetFlow provides, I still often have enough data and at a much lower cost.

These are just my experiences and are specific to our environment, I am sure this doesn't apply to everybody.

MVP
MVP

Excellent topic...while we have it I have not had the time to fully explore it.

Level 13

I'd have to say that this pretty much mirrors our current environment also.

MVP
MVP

we have just upgraded to NTA4 and are exploring what we can do with flow.

Level 12

I think it is great that you are pointing out some of the possible uses for Netflow.  So many of the people I work with only use it for reactionary troubleshooting of network congestion.

Level 13

What are you using for a SIEM that's less expensive than NetFlow?

Level 21

We are using SolarWinds LEM.  It not necessarily less from a direct product cost perspective; however, if you take everything else into consideration such as storage,system, network footprint and then factor in that you also have SIEM which we need anyway then you begin to see a significant savings.

Level 12

@Jfrazier Thanks. Hope this article would trigger you to explore more on NetFlow.

About the Author
CCIE Data Center #46006. I am passionate IT professional who splits the work hours as a Datacenter Architect and a Network Security Specialist. Yes, I enjoy this double personality professional life.