16 Replies Latest reply on Dec 1, 2016 8:01 AM by ken_cohen

    Reporting on NetFlow data

    ken_cohen

      I've been asked by our Security team for reporting on our internet connections.  They want who's going where and for what content.

       

      First, I would think that a report like this should already be available.  It seems like a no-brainer to me 

       

      However, the canned web console reports don't have the ability to filter to specific interfaces, and Report Writer doesn't have any NetFlow data sources.  I find this ridiculous at best, and aggravating at worst.

       

      What's the point of having all the NetFlow data if I can't report on it?

       

      Has anybody else faced this situation?

        • Re: Reporting on NetFlow data
          rschroeder

          Your security team needs to have a content filter in place, a proxy solution, to get this information in the granular format they desire.  The should already be filtering, and be aware of every request that hits your firewall--both from outside and from inside.

           

          The simple solution is to place physical network taps between your internal network and your firewall, and also between the firewall and the Internet. Then send that traffic to a service like Web Sense (or any of many other content filters).  Capturing that data and sending it to their SIEM (like Splunk or SW's product or any other SIEM) will enable them to see what they need between your organization and the Internet.

           

          If they need greater detail, look into something like Gigamon's product line to accomplish this.  Gigamon will send ALL NetFlow data and parse it out to MANY different kinds of reports.  It's not inexpensive.

           

          Regarding what Orion can do, you can capture all the syslog data from your firewalls and send it to your SIEM; this gets them the north-south data, but none of the east-west data.  For that in-house traffic you can capture NetFlow traffic at your L3 interfaces--but you'll still miss the Layer 2 E-W traffic because NetFlow is only captured where traffic changes subnets.  Traffic within a subnet is not captured.  This is where something like Gigamon is required.

           

          To get a lightweight idea of what traffic is associated with the Internet--using an Orion suite product--you must have NetFlow Traffic Analyzer.  It can provide sampled data that can show trends, but I don't feel it's adequate for security purposes.  It can show you information about flows where it recognizes

           

          • The applications being used (it does NOT recognize all apps)
          • Conversations between a source & destination end point
          • Conversations where countries involved are identified by IP address ranges
          • Conversations associated with specified end points
          • Conversations with specific receivers and transmitters
          • Flows associated with defined / recognized IP Groups
          • Protocols involved in conversations
          • Type of Service
          • BGP flows

           

          This sounds like a lot, and it is. But it's not everything a Security person needs. Sure, it shows a PC going to Yahoo or Google, but does NOT show what's being searched for or accessed--only the protocols, the source & destination addresses. And Security folks need the extra detail--it's why they use proxies and content filters.

           

          East-West traffic is important to capture for watching malicious flows, seeing what malware is touching every address, and flagging it for investigation/remediation. NetFlow won't do that.

           

          NTA can show some quickly useful information for a Network Analyst/Engineer--what are the Top 25 Destinations by Domain, or the Top 25 Traffic Sources, or the Top Applications, or the Top Conversations. But the detail needed by Security is not there. And this is sampled data--not 100% of the data is present. Hey--it's NetFlow, designed for troubleshooting flows, not for reporting security events. Sure, some of its data can be used by Security, but much of what they need will have to be collected from taps (or spans) and syslog information sent to a SIEM that can sort & analyze automatically--and report as needed.

           

          Make sure your Security team understands the limitations of what NetFlow can do for them, and where it won't work, and what level of detail it cannot provide.

            • Re: Reporting on NetFlow data
              ken_cohen

              Thank you for the response – I really appreciate it.

               

              I differ, however, in that we’ve laid out significant $$$ for a tool that is already gathering the data that Security wants, and I feel I shouldn’t have to jump through even more hoops, adding new infrastructure, etc. to be able to collect / report on data that NTA has already gathered.  I just want to report on that data, not re-invent a wheel.

               

              I’ve been griping at SW for some time now about this sort of thing.  They love to introduce new features, but never go back and fix older issues.  Reporting based on business hours / time zones, anyone?  That’s been around for years and still no progress.

               

              Much as I love SolarWinds, if I don’t see some movement along these lines and soon, I’m going to have to dump it and move on to a tool that fills my needs.  I shouldn’t have to fight the tools, and I’m very unhappy that once again SW is ALMOST there, but doesn’t quite do the job I was told it would do. I don’t think it’s asking too much to be able to report on collected data.  I need a viable solution with the tool we’ve already bought, or it’s goodbye.

               

              Kenneth W. Cohen

              Network Analyst

              Global Technical Services, IT

              Office: 470-448-5870

              Mobile: 678-428-9875

               

               

              kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

              Halyardhealth.com

                • Re: Reporting on NetFlow data
                  mesverrum

                  I think rschroeder's point is that netflow as a protocol doesn't support the granularity a security team needs.  It was not intended from it's origins back at Cisco in the 90's to be a detailed tool for analyzing specific data as much as it is a ballpark tool for measuring which way the wind is blowing.  In many cases you aren't even grabbing info from every packet, just a sampled percentage of them.  There have been improvements to that since then such as NBAR2 (which NTA does support) but even that is still not generally going to be as good as setting up an analysis of the web content filter/firewalls themselves using a proper SIEM (which Solarwinds offers in their LEM product, also Splunk is popular for that kind of thing).

                   

                  With that said, the best way to build reports from Solarwinds of your netflow based data is just to drill down through the netflow pages to show the data you want in the report (presumably both in and outbound traffic across your WAN interface for the past 24 hours or something like that) and bookmark that, all your filter settings are embedded in the URL.  Go to the report scheduler and just tell it to send a webpage, put that url in there and every morning your security guys can check it out in their inbox.  Presumably instead of top 10's and such though for security purposes you would need something like top 10,000 conversations otherwise the little oddball things that security tends to be looking for would never even make it to the list. 

                  Likewise you could go to the endpoint details view for servers that store sensitive data, but because netflow is usually only being captured at the edges you would only see if someone connected from outside directly to the server, which should not happen except to DMZ servers.  If the attacker uses a different host inside the same subnet as a jump box then that wouldn't show up as a netflow from your router at all because, as mentioned, east/west type traffic is relatively uncommon to capture and most distribution/edge gear doesn't support the protocol it if you wanted to.

                   

                  Also, since it sounds like you may not have heard, business hours reporting had been available in the old Reportwriter for a long time and was added to the web based reporting in June.

                   

                  -Marc Netterfield

                      Loop1 Systems: SolarWinds Training and Professional Services

                    • Re: Reporting on NetFlow data
                      ken_cohen

                      I’m well aware, but this still doesn’t cut the mustard for international companies like mine.  If you try build a report where the time crosses midnight it fails.  I ended up building dual staged reports for these sites where the first ends at midnight and the next starts at midnight.  It’s very clumsy and difficult to interpret for an entire shift, but it’s the only way I could automate it.

                       

                      Kenneth W. Cohen

                      Network Analyst

                      Global Technical Services, IT

                      Office: 470-448-5870

                      Mobile: 678-428-9875

                       

                       

                      kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

                      Halyardhealth.com

                        • Re: Reporting on NetFlow data
                          mesverrum

                          Crossing the midnight threshold makes it a bit more complicated, but still doable.  In this case lets assume I am monitoring something 12 hours east of the Solarwinds server time zone, so my server is in California and I'm building a report for a system in Dubai.  I would just tweak it like below.  Solarwinds in general does have a stack of problems with having pollers in multiple timezones and they need to fix that, but for now it seems the best work around is just to set all your pollers to a single time zone regardless of where they live.  Anyway I digress.  This would allow you to take those two reports breaking at midnight and combine them back into one that covers the spread.

                           

                          In most cases that I have run into there may be things that are hard-ish to do in solarwinds, but it is an open enough platform that I can almost always rig up some kind of solution/modification to accomplish any wish list item.

                           

                          -Marc Netterfield

                              Loop1 Systems: SolarWinds Training and Professional Services

                  • Re: Reporting on NetFlow data
                    rschroeder

                    I think you'll find that NetFlow does not provide the level of detail that your Security team requires, and that only a proxy / content filter--either separate from a firewall or built into it--can accomplish.

                     

                    NetFlow provides source & destination IP addresses that are participating in conversations which travers L3 Netflow interfaces, but not destination URLs and details about which pages are hit, and what kind of page they are (based on categories--Sports, Political, Religion, etc.).

                     

                    You can use NetFlow to find which IP addresses are consuming WAN bandwidth, what protocols are being used, and what the percentage of network use is for any of them.

                     

                    It's the right tool for Network Admins, and it can be helpful for some security tasks if all the traffic passes through L3 interfaces.  But getting East-West data, or all the North-South data, isn't going to happen without huge investments in infrastructure like taps and Gigamon infrastructure, proxy filters, firewalls, a properly sized SIEM, etc.  NetFlow only has a small portion of the data Security requires.

                     

                    But I sympathize with you--it would be nice to share the NetFlow data your equipment can collect with Security, but you have to have something like NTA to do that.

                      • Re: Reporting on NetFlow data
                        ken_cohen

                        OK - I think everyone responding kind of missed the point.  I shouldn't have mentioned Security in the original question as we all went off on a tangent on what a Security group needs.  That discussion is irrelevant to the actual thrust of the question, which is reporting on NetFlow data and not what Security does or doesn't need.

                         

                        So, back to the original question - What do I have to do report on NetFlow data in the same manner as I do with NPM collected data?  The data is there, I just want to build reports on it.

                          • Re: Reporting on NetFlow data
                            rschroeder

                            That's a good question, and the answer is beyond me.  You can send Netflow data at your NPM server, but I don't see how NPM can work with it until NTA is installed.  I DO understand your concept:  NPM monitors performance, Netflow is all about performance, NPM therefore "should" be able to build useful reports from Netflow data it receives. 

                             

                            But I think that's why NTA was built--because NPM can't do what's desired with the Netflow data.

                             

                             

                             

                            Routers & L3 switches that capture and forward Netflow information have the ability to provide you Netflow data, but it's in data format--not useful for reporting.  And they do it on a per-router basis, not something you can report about for all your devices with NPM.

                             

                            Let's say you've got a Cisco IOS router set up for Netflow exporting to NPM.  Without NTA you can't see the data, NPM has no idea of what to do with it.  So you SSH into the router and issue the command "show ip cache flow".  What comes out is a lot of raw stats, followed by individual conversation flow stats that show packet counts, src addr and dst addr info:

                             

                             

                            Now that you have this manually from one router, you can't do a lot with it other than calculate percentages of traffic each flow makes up.  But you've only gathered the info for one router--not your entire organization's count of routers.

                             

                            In the section showing src & dst addresses, you can't tell what the content is, nor the amount of traffic.  It's why NTA or one of its competitors is required:

                             

                            You could enable ip flow top-talkers on each router, and then get some idea about who's using the most bandwidth.  But this remains a manual and tedious per-device-process.

                             

                            Suppose you could run a script with NCM that captured this info from each router on a regular basis.  What would you do with it, except try manually porting it to Excel or a database and kluge together your own custom reporting solution.   You'd still need a management tool built specifically for this data--like NTA--to get useful (and beautiful) data from Netflow.

                             

                            I feel your pain--not wanting to buy NTA when you have NPM is understandable.  One could easily imagine NTA being a simple series of reports within NPM.  Which is how it works when you integrate NTA with NPM.

                             

                            Here's hoping I'm mistaken.  Maybe some uber-Geeky Thwack-meister has just what you want, and can help you fill your needs without NTA.  If that happens, please share so the rest of the folks can understand what you end up doing.  Maybe they can leverage some of your process in the future, especially if they're too small to be able to afford NTA. 

                             

                            Because your idea's a reasonable one, IMHO.

                              • Re: Reporting on NetFlow data
                                ken_cohen

                                I have NTA and it works very well.  I just want to report on the data it collects.

                                 

                                Kenneth W. Cohen

                                Network Analyst

                                Global Technical Services, IT

                                Office: 470-448-5870

                                Mobile: 678-428-9875

                                 

                                 

                                kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

                                Halyardhealth.com

                                  • Re: Reporting on NetFlow data
                                    rschroeder

                                    Ah, that makes it so very different.    Please accept my apologies for not understanding you have NTA.

                                     

                                    NTA does not provide user names (like Active Directory authenticated users) and URL information (like destination web pages).  Only bandwidth quantities, src & dst addresses, domains, etc.

                                     

                                    The information you seek requires a proxy filter and/or  tap and SIEM solution, along with access to AD for user information.  Netflow doesn't include that kind of data at all, as you could see from the previous screen shots I shared.

                                      • Re: Reporting on NetFlow data
                                        ken_cohen

                                        Beg to differ – the data that I want to report on has already been collected.  As I’ve said several times now, please forget that the intent is to send a report to Security.  That is completely irrelevant.

                                         

                                        I can pull up and set parameters on the NTA console and go back in time as far as I want with collected NetFlow data.  That fine, but it doesn’t help for automated report generation (and let me repeat here) for the data that has been collected and stored.

                                         

                                        The problem with the Web Report engine in the console is it does not let me filter to specific interfaces.  It gets me the data I want to report on, but there’s a zillion interfaces we’re collecting from which makes the report way too long.  I would do it in Report Writer on the server, but Report Writer doesn’t include any NetFlow data as sources to report against.

                                         

                                        That is the conundrum – the data is there but I cannot generate a report against it.

                                         

                                        I don’t think I can make my issue any more clear.  It doesn’t make sense to me that I cannot generate a report (the way I want to distribute it, just like I’m already doing with NPM data) on NetFlow data that is just sitting in the DB.

                                         

                                        Thanks for letting me vent – I’m very frustrated by this, and a bit embarrassed as well as I’m the one who recommended SW in the first place.

                                         

                                        Kenneth W. Cohen

                                        Network Analyst

                                        Global Technical Services, IT

                                        Office: 470-448-5870

                                        Mobile: 678-428-9875

                                         

                                         

                                        kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

                                        Halyardhealth.com

                                          • Re: Reporting on NetFlow data
                                            rschroeder

                                            Yep, you're right.  My bad.  I'm dealing with Security Teams steadily, and I'm sidetracked by that.  I should've remained silent.  I'm sorry.

                                             

                                            I focused on your mentioning that your security team wants to know who's going where, and for what content, and to me that means AD user names and URL destinations--which are captured by proxy or content filters--because I didn't think Netflow captured user names and URLs.  I was wrong.  Firewalls from Cisco, Dell, and Palo Alto that support Netflow CAN capture user names.

                                             

                                            You mentioned your canned reports don't have that info (they don't show user names) and they don't offer an ability to report by interfaces.  That should've told me you had NTA, but I thought you were trying to only use NPM; again, my bad, and I apologize.

                                             

                                            I've built a real-time display that shows the top sources by domain, and top destinations by domain, which first filters out my company's domains from the list.  It shows how much traffic goes to any given domain, from any given domain, but it's not in report format like you'd need.  Nor does it show user name or destination URL.

                                             

                                            I'm open to all others' suggestions.  You're right--Netflow captures URLs and user names, and it seems NTA should be able to report on them just as a proxy or content filter or firewall can.

                                             

                                            You can Google the topic and see there are several solutions, including Plixer, Cisco ISE, and Microsoft Domain Name Services--all of which have methods of displaying user names when the traffic is captured at a compatible firewall.

                                             

                                            I've reviewed the Netflow Version 9 tables and offerings; I don't see a way they can provide user name in reports unless a firewall is doing the reporting, coordinating with AD.

                                             

                                            Do you see a place in your NTA output that shows users' names and destination URL's?  If so, please share it.  Perhaps Thwack members can provide a way to get it into report form for you.

                                              • Re: Reporting on NetFlow data
                                                ken_cohen

                                                No, I don’t see that, but that doesn’t matter to me as all I want to report on are source / destination IPs and port numbers going over our internet-facing interfaces around our global enterprise network.  The data is there, I just can’t report against it.

                                                 

                                                If this simple situation cannot be resolved and I am forced to get another tool that will do what I need, then it’s going to be goodbye to SW.  I truly cannot fathom why I should have to stand up another tool when the tool I already invested in has the data I want to see but cannot report on it.

                                                 

                                                Kenneth W. Cohen

                                                Network Analyst

                                                Global Technical Services, IT

                                                Office: 470-448-5870

                                                Mobile: 678-428-9875

                                                 

                                                 

                                                kenneth.cohen@hyh.com<mailto:kenneth.cohen@hyh.com>

                                                Halyardhealth.com

                                              • Re: Reporting on NetFlow data
                                                mesverrum

                                                What you are describing is doable from the web report tool.  Just take the built in top 100 conversations report and edit the data source part to only include interfaces with name like such and such, or if you have an interface custom property you can use that.

                                                 

                                                Then just add in these columns to the table and group by the interface fullname

                                                 

                                                 

                                                Then you end up with something roughly like this

                                                 

                                                 

                                                 

                                                 

                                                -Marc Netterfield

                                                    Loop1 Systems: SolarWinds Training and Professional Services

                                    • Re: Reporting on NetFlow data
                                      darragh.delaney

                                      Hi ken_cohen

                                      We develop a product called LANGuardian which uses packet information as a data source. It extracts certain metadata from HTTP headers like domain names and URI and associates this with user info from AD if needed. The data we capture can be displayed within SolarWinds views and you can see a demo of this in action at the link below. I also included a video at the end of the post which uses the same demo.

                                       

                                      http://demo2.netfort.com/Orion/SummaryView.aspx?AccountID=guest&viewid=31

                                       

                                      We also run IDS engines in parallel so you can watch out for suspicious traffic. I am also not convinced that content filters are the solution. I have picked up on an increase in Hola use on some networks where users were trying to get around web filters. This just introduces a whole lot of other issues for the network\security team. I covered this and other topics on my blog.

                                       

                                      Working with metadata is like having a car with a manual and automatic option. Pop into manual when you want to get technical, have a bit of fun and look at packet content. Slide into automatic when you want an easier life and just look at nice graphs and traffic volumes.

                                       

                                      Darragh