Reporting on NetFlow data

Your security team needs to have a content filter in place, a proxy solution, to get this information in the granular format they desire.  The should already be filtering, and be aware of every request that hits your firewall--both from outside and from inside.

The simple solution is to place physical network taps between your internal network and your firewall, and also between the firewall and the Internet. Then send that traffic to a service like Web Sense (or any of many other content filters).  Capturing that data and sending it to their SIEM (like Splunk or SW's product or any other SIEM) will enable them to see what they need between your organization and the Internet.

If they need greater detail, look into something like Gigamon's product line to accomplish this.  Gigamon will send ALL NetFlow data and parse it out to MANY different kinds of reports.  It's not inexpensive.

Regarding what Orion can do, you can capture all the syslog data from your firewalls and send it to your SIEM; this gets them the north-south data, but none of the east-west data.  For that in-house traffic you can capture NetFlow traffic at your L3 interfaces--but you'll still miss the Layer 2 E-W traffic because NetFlow is only captured where traffic changes subnets.  Traffic within a subnet is not captured.  This is where something like Gigamon is required.

To get a lightweight idea of what traffic is associated with the Internet--using an Orion suite product--you must have NetFlow Traffic Analyzer.  It can provide sampled data that can show trends, but I don't feel it's adequate for security purposes.  It can show you information about flows where it recognizes

• The applications being used (it does NOT recognize all apps)
• Conversations between a source & destination end point
• Conversations where countries involved are identified by IP address ranges
• Conversations associated with specified end points
• Conversations with specific receivers and transmitters
• Flows associated with defined / recognized IP Groups
• Protocols involved in conversations
• Type of Service
• BGP flows

This sounds like a lot, and it is. But it's not everything a Security person needs. Sure, it shows a PC going to Yahoo or Google, but does NOT show what's being searched for or accessed--only the protocols, the source & destination addresses. And Security folks need the extra detail--it's why they use proxies and content filters.

East-West traffic is important to capture for watching malicious flows, seeing what malware is touching every address, and flagging it for investigation/remediation. NetFlow won't do that.

NTA can show some quickly useful information for a Network Analyst/Engineer--what are the Top 25 Destinations by Domain, or the Top 25 Traffic Sources, or the Top Applications, or the Top Conversations. But the detail needed by Security is not there. And this is sampled data--not 100% of the data is present. Hey--it's NetFlow, designed for troubleshooting flows, not for reporting security events. Sure, some of its data can be used by Security, but much of what they need will have to be collected from taps (or spans) and syslog information sent to a SIEM that can sort & analyze automatically--and report as needed.

Make sure your Security team understands the limitations of what NetFlow can do for them, and where it won't work, and what level of detail it cannot provide.

Thank you for the response – I really appreciate it.

I differ, however, in that we’ve laid out significant $$$ for a tool that is already gathering the data that Security wants, and I feel I shouldn’t have to jump through even more hoops, adding new infrastructure, etc. to be able to collect / report on data that NTA has already gathered. I just want to report on that data, not re-invent a wheel.

I’ve been griping at SW for some time now about this sort of thing. They love to introduce new features, but never go back and fix older issues. Reporting based on business hours / time zones, anyone? That’s been around for years and still no progress.

Much as I love SolarWinds, if I don’t see some movement along these lines and soon, I’m going to have to dump it and move on to a tool that fills my needs. I shouldn’t have to fight the tools, and I’m very unhappy that once again SW is ALMOST there, but doesn’t quite do the job I was told it would do. I don’t think it’s asking too much to be able to report on collected data. I need a viable solution with the tool we’ve already bought, or it’s goodbye.

Kenneth W. Cohen

Network Analyst

Global Technical Services, IT

Office: 470-448-5870

Mobile: 678-428-9875

kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

Halyardhealth.com

I think you'll find that NetFlow does not provide the level of detail that your Security team requires, and that only a proxy / content filter--either separate from a firewall or built into it--can accomplish.

NetFlow provides source & destination IP addresses that are participating in conversations which travers L3 Netflow interfaces, but not destination URLs and details about which pages are hit, and what kind of page they are (based on categories--Sports, Political, Religion, etc.).

You can use NetFlow to find which IP addresses are consuming WAN bandwidth, what protocols are being used, and what the percentage of network use is for any of them.

It's the right tool for Network Admins, and it can be helpful for some security tasks if all the traffic passes through L3 interfaces.  But getting East-West data, or all the North-South data, isn't going to happen without huge investments in infrastructure like taps and Gigamon infrastructure, proxy filters, firewalls, a properly sized SIEM, etc.  NetFlow only has a small portion of the data Security requires.

But I sympathize with you--it would be nice to share the NetFlow data your equipment can collect with Security, but you have to have something like NTA to do that.

I think rschroeder​​'s point is that netflow as a protocol doesn't support the granularity a security team needs.  It was not intended from it's origins back at Cisco in the 90's to be a detailed tool for analyzing specific data as much as it is a ballpark tool for measuring which way the wind is blowing.  In many cases you aren't even grabbing info from every packet, just a sampled percentage of them.  There have been improvements to that since then such as NBAR2 (which NTA does support) but even that is still not generally going to be as good as setting up an analysis of the web content filter/firewalls themselves using a proper SIEM (which Solarwinds offers in their LEM product, also Splunk is popular for that kind of thing).

With that said, the best way to build reports from Solarwinds of your netflow based data is just to drill down through the netflow pages to show the data you want in the report (presumably both in and outbound traffic across your WAN interface for the past 24 hours or something like that) and bookmark that, all your filter settings are embedded in the URL.  Go to the report scheduler and just tell it to send a webpage, put that url in there and every morning your security guys can check it out in their inbox.  Presumably instead of top 10's and such though for security purposes you would need something like top 10,000 conversations otherwise the little oddball things that security tends to be looking for would never even make it to the list. 

Likewise you could go to the endpoint details view for servers that store sensitive data, but because netflow is usually only being captured at the edges you would only see if someone connected from outside directly to the server, which should not happen except to DMZ servers.  If the attacker uses a different host inside the same subnet as a jump box then that wouldn't show up as a netflow from your router at all because, as mentioned, east/west type traffic is relatively uncommon to capture and most distribution/edge gear doesn't support the protocol it if you wanted to.

Also, since it sounds like you may not have heard, business hours reporting had been available in the old Reportwriter for a long time and was added to the web based reporting in June.

-Marc Netterfield

    Loop1 Systems: SolarWinds Training and Professional Services

•                 LinkedIN: Loop1 Systems
•                 Facebook: Loop1 Systems
•                 Twitter: @Loop1Systems

Hi ken_cohen​

We develop a product called LANGuardian which uses packet information as a data source. It extracts certain metadata from HTTP headers like domain names and URI and associates this with user info from AD if needed. The data we capture can be displayed within SolarWinds views and you can see a demo of this in action at the link below. I also included a video at the end of the post which uses the same demo.

http://demo2.netfort.com/Orion/SummaryView.aspx?AccountID=guest&viewid=31

We also run IDS engines in parallel so you can watch out for suspicious traffic. I am also not convinced that content filters are the solution. I have picked up on an increase in Hola use on some networks where users were trying to get around web filters. This just introduces a whole lot of other issues for the network\security team. I covered this and other topics on my blog.

Working with metadata is like having a car with a manual and automatic option. Pop into manual when you want to get technical, have a bit of fun and look at packet content. Slide into automatic when you want an easier life and just look at nice graphs and traffic volumes.

Darragh

OK - I think everyone responding kind of missed the point.  I shouldn't have mentioned Security in the original question as we all went off on a tangent on what a Security group needs.  That discussion is irrelevant to the actual thrust of the question, which is reporting on NetFlow data and not what Security does or doesn't need.

So, back to the original question - What do I have to do report on NetFlow data in the same manner as I do with NPM collected data?  The data is there, I just want to build reports on it.

I’m well aware, but this still doesn’t cut the mustard for international companies like mine. If you try build a report where the time crosses midnight it fails. I ended up building dual staged reports for these sites where the first ends at midnight and the next starts at midnight. It’s very clumsy and difficult to interpret for an entire shift, but it’s the only way I could automate it.

Kenneth W. Cohen

Network Analyst

Global Technical Services, IT

Office: 470-448-5870

Mobile: 678-428-9875

kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

Halyardhealth.com

That's a good question, and the answer is beyond me.  You can send Netflow data at your NPM server, but I don't see how NPM can work with it until NTA is installed.  I DO understand your concept:  NPM monitors performance, Netflow is all about performance, NPM therefore "should" be able to build useful reports from Netflow data it receives. 

But I think that's why NTA was built--because NPM can't do what's desired with the Netflow data.

Routers & L3 switches that capture and forward Netflow information have the ability to provide you Netflow data, but it's in data format--not useful for reporting.  And they do it on a per-router basis, not something you can report about for all your devices with NPM.

Let's say you've got a Cisco IOS router set up for Netflow exporting to NPM.  Without NTA you can't see the data, NPM has no idea of what to do with it.  So you SSH into the router and issue the command "show ip cache flow".  What comes out is a lot of raw stats, followed by individual conversation flow stats that show packet counts, src addr and dst addr info:

Now that you have this manually from one router, you can't do a lot with it other than calculate percentages of traffic each flow makes up.  But you've only gathered the info for one router--not your entire organization's count of routers.

In the section showing src & dst addresses, you can't tell what the content is, nor the amount of traffic.  It's why NTA or one of its competitors is required:

You could enable ip flow top-talkers on each router, and then get some idea about who's using the most bandwidth.  But this remains a manual and tedious per-device-process.

Suppose you could run a script with NCM that captured this info from each router on a regular basis.  What would you do with it, except try manually porting it to Excel or a database and kluge together your own custom reporting solution.   You'd still need a management tool built specifically for this data--like NTA--to get useful (and beautiful) data from Netflow.

I feel your pain--not wanting to buy NTA when you have NPM is understandable.  One could easily imagine NTA being a simple series of reports within NPM.  Which is how it works when you integrate NTA with NPM.

Here's hoping I'm mistaken.  Maybe some uber-Geeky Thwack-meister has just what you want, and can help you fill your needs without NTA.  If that happens, please share so the rest of the folks can understand what you end up doing.  Maybe they can leverage some of your process in the future, especially if they're too small to be able to afford NTA. 

Because your idea's a reasonable one, IMHO.

I have NTA and it works very well. I just want to report on the data it collects.

Kenneth W. Cohen

Network Analyst

Global Technical Services, IT

Office: 470-448-5870

Mobile: 678-428-9875

kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

Halyardhealth.com

Ah, that makes it so very different.    Please accept my apologies for not understanding you have NTA.

NTA does not provide user names (like Active Directory authenticated users) and URL information (like destination web pages).  Only bandwidth quantities, src & dst addresses, domains, etc.

The information you seek requires a proxy filter and/or  tap and SIEM solution, along with access to AD for user information.  Netflow doesn't include that kind of data at all, as you could see from the previous screen shots I shared.