23 Replies Latest reply on Jan 25, 2016 9:34 AM by Andy McBride

    Replicated NetFlow

    BryanBecker

      Since we send the NetFlow streams to 2 different systems I wondered if Orion can handle streams sent to it from a replicator device?  For instance we also have Arbor that can replicate the data and forward it on. We are trying this without any luck.

      Basically we want to send 1 stream and have Arbor forward that to us as well.  The data should be good still.

      BB

        • Re: Replicated NetFlow
          Yann

          Hi,

          You might want to give a quick try to Samplicator: http://www.switch.ch/network/downloads/tf-tant/samplicator/

          This small program receives UDP datagrams on a given port, and resends those datagrams to a specified set of receivers.

          HTH,

          Yann

            • Re: Replicated NetFlow
              firehawk_350

              Does Orion NCM not have the ability to forward the Netflow to the 2nd system such as QRadar

                • Re: Replicated NetFlow
                  firehawk_350

                  I meant Orion Netflow Analyzer

                    • Re: Replicated NetFlow
                      kweise

                      firehawk_350,

                      As far as I know, the NTA is just a netflow collector.  It doesn't have any way to forward the netflow streams it receives to your QRadar box.  You can configure netflow (at least on Cisco boxes) to export to multiple destinations.  Just add another "ip flow-export destination" statement.

                      Hope this helps.

                        • Re: Replicated NetFlow
                          Andy McBride

                          kweise is correct, you can set you exporter (router or such) to export to a second IP. NTA does not do this.

                          • Re: Replicated NetFlow
                            BryanBecker

                            I think we all agree that NTA is a collector only.  The fact is more and more tools need Netflow data.  Cisco has a 2 destination limit and just adding a 2nd destination isn't always the right answer.  If you have over 2 collection machines then your out of luck.  Netflow can be a WAN hog so if you need to send it to 2 systems across the WAN your doubling the exact same traffic.  In a hub and spoke network, like mine, that's not an attractive solution.  Since, in our network, the Netflow receivers like NTA sit in the same network it would make sense to send 1 flow over the WAN and it can be replicated on the LAN.

                            So add me to the list that would like to see NTA be a flow redirector/replicator.

                            BB

                              • Re: Replicated NetFlow
                                warbird

                                I feel your pain with the slower spokes in a network.  In my network, we have some remote sites serviced by frame relay satellite uplinks.  Most of these sites have only 1 or 2 T1s, with all of the latency that comes with satellite.  In these cases, we had to set up a cheapo flow collector system at each remote site.  This is nice because you can copy the remote site flow data to your main flow repository in your core network late at night.  Not sure if Orion would be able to import and aggregate flows in this manner, tho...?  I apologize if this isn't helping, I've just delved into NetFlow with Orion and am still learning.

                                Also, I do not understand how having Orion be a flow replicator is better?  You would be using Orion to do what I and others have suggested, thereby creating the same WAN traffic problem?  And would this not also cause more load to an already busy polling engine?

                                  • Re: Replicated NetFlow
                                    MelloDL

                                    War Bird was exactly right in his comments and suggestions. There are a number of free, open-source "fan-out" solutions that can be installed to a relatively low-end server running some version of Linux.

                                    Even though you can configure a Cisco router to send duplicate copies of NetFlow streams to two different destinations, you would be foolish to do this. Why send duplicate (or triplicate) copies of the same NetFlow streams across precious WAN links.

                                    You should configure each of your routers to send their NetFlow streams to a single "fan-out" box which in turn can redistribute copies of those NetFlow streams to each and every NetFlow analysis tool that you might wish to use.

                                    We collect NetFlow streams from about 250 routers and redirect those streams to NetQoS Reporter Analyzer, Plixer Scrutinizer, Solarwinds NetFlow Traffic Analyzer. However because of the high cost and licensing per interface of NetQoS Reporter Analyzer, we also have implemented a solution in which we use different port designations in the router configurations to act as policy proxies so that the fan-out box will forward all streams to the NetFlow Traffic Analyzer and Scrutinizer software but only some streams to the Reporter Analyzer software.

                                    You can do a Google search to easily find free, open source software to run the 'fan-out' process; and it doesn't take a 'rocket scientist' to figure out the Linux OS on which to run the 'fan-out' service.

                                    http://linux.softpedia.com/get/System/Networking/flowtools-13642.shtml

                                    Hi, if one of your flow repository boxen is linux, then I suggest you use flow-fanout.  Depending on how your repository is currently setup (ie. what software you are currently using to capture the flows), this is very simple to do.  The man page for flow-fanout has good information.  This is what I am using to send flows to my Orion instance. The config file to setup with your flow exporters and receivers is as follows.  I added in the comments in the parentheses to help explain it.  If you are unfamiliar with *nix, the comment lines begin with #.  Then again, if you are unfamiliar with linux, this prolly isn't for you.  ;-)  ***

                                    [imroot@yoursys ~]$ less /etc/sysconfig/flow-fanout
                                    # ftOptions: Use "-s" to enable spoofing
                                    ftOptions=-s
                                    # ftLocalAddress: defaults to all interfaces
                                    #
                                    # (This is the IP your exporters are sending flows to)
                                    ftLocalAddress=xxx.xxx.x.xx
                                    # ftDestinations: defaults to "0/127.0.0.1/9995"
                                    #
                                    # (These are where you are sending the flows to.  Must include home to keep copy of flows here.
                                    # You can have as many as your system will handle.)
                                    #
                                    # ftDestinations="0/127.0.0.1/9995"
                                    ftDestinations="0/127.0.0.1/9995 0/xxx.xxx.xxx.xxx/9995"
                                    # ftRemoteSenders: defaults to "0:9996"; may specify multiple separated by space,
                                    #
                                    # (These are your flow exporters.  You can have as many as your system will handle)
                                    ftRemoteSenders="\
                                    yyy.yyy.y.y:9982 yyy.yyy.yyy.y:9983 \
                                    yyy.yyy.y.yyy:9985"

                                    ***

                                    Anytime you make changes to this config file, you'll need to bounce the flow-fanout service.

                                    [imroot@yoursys ~]# service flow-fanout restart

                        • Re: Replicated NetFlow
                          warbird

                          Hi,

                          If one of your flow repository boxen is linux, then I suggest you use flow-fanout.  Depending on how your repository is currently setup (ie. what software you are currently using to capture the flows), this is very simple to do.  The man page for flow-fanout has good information.  This is what I am using to send flows to my Orion instance.

                          The config file to setup with your flow exporters and receivers is as follows.  I added in the comments in the parentheses to help explain it.  If you are unfamiliar with *nix, the comment lines begin with #.  Then again, if you are unfamiliar with linux, this prolly isn't for you.  ;-) 

                          ***

                          [imroot@yoursys ~]$ less /etc/sysconfig/flow-fanout

                          # ftOptions: Use "-s" to enable spoofing

                          ftOptions=-s

                          # ftLocalAddress: defaults to all interfaces
                          #
                          # (This is the IP your exporters are sending flows to)

                          ftLocalAddress=xxx.xxx.x.xx

                          # ftDestinations: defaults to "0/127.0.0.1/9995"
                          #
                          # (These are where you are sending the flows to.  Must include home to keep copy of flows here.
                          # You can have as many as your system will handle.)
                          #
                          # ftDestinations="0/127.0.0.1/9995"

                          ftDestinations="0/127.0.0.1/9995 0/xxx.xxx.xxx.xxx/9995"

                          # ftRemoteSenders: defaults to "0:9996"; may specify multiple separated by space,
                          #
                          # (These are your flow exporters.  You can have as many as your system will handle)

                          ftRemoteSenders="\
                          yyy.yyy.y.y:9982 yyy.yyy.yyy.y:9983 \
                          yyy.yyy.y.yyy:9985"

                          ***

                          Anytime you make changes to this config file, you'll need to bounce the flow-fanout service.

                          [imroot@yoursys ~]# service flow-fanout restart

                          • Re: Replicated NetFlow
                            banditone

                            Any new developments from Solarwinds on this? Perhaps NTA is now able to be setup to forward on 'fan-out' data after it receives it now? The last update on this was 2009. It's 2015 now, we put a man on the moon. Surely this option now exists?

                              • Re: Replicated NetFlow
                                michal.hrncirik

                                Non from SolarWinds NTA side. And it unlikely will be from our side in the near future. Our NTA is primarily NetFlow collector and doesn't want to become a central router for incoming NetFlow streams (that's task for other - ideally HW vendors). NTA wants to be a better bandwidth monitoring tool and get more application visibility in: "what, who, when" question.

                                More than forwarding flows somewhere else, tell us why you'd need to forward it (better analysis, reporting, etc.) so we can focus on making NTA your primary and only tool for "flow" based analysis.

                                 

                                thanks,

                                Michal

                                  • Re: Replicated NetFlow
                                    sja

                                    Michal

                                     

                                    I also don't see the point of forwarding flows up "to a super collector" :-)

                                    But see point and value of forwarding "events" from the down stream collector in form of syslog and traps

                                    that link to the event to NPM and LEM or 3 party SIEM.


                                    /SJA

                                     

                                     

                                    • Re: Replicated NetFlow
                                      banditone

                                      michal.hrncirik  - 

                                      Thank you for the concise and informative answer. Exactly what I was needing. Regards, Shaun.

                                      • Re: Replicated NetFlow
                                        RichardLetts

                                        Orion's' unnerving predilection for deleting data whenever it feels like it (e.g. if we delete a node that has sent us netflow data) rather than when we no-longer need the data or being able to easily restore a single node (undelete) means we could not use Solarwinds as a reliable collector for billing purposes [commodity transit internet traffic]

                                         

                                        We've some nodes in our database set to be unmanaged until 2025 just because we don't want to lose the historical availability data tied to them...

                                         

                                        (we run a reflector on one of our linux systems and fan-out the UDP packets to every appliance that is interested in handling them... from accounting systems through to the troubleshooting systems).

                                          • Re: Replicated NetFlow
                                            michal.hrncirik

                                            You're right Richard, our tools are not primarily designed for billing purposes (or "re-play" simulation) therefore data deletion mechanism is more oriented for data consistency and performance. I'm curious about your nodes where you want to keep in the database. What exactly you plan to do with the data? It's there for random check of the problem, looking for some pattern, etc? Orion typically aggregates historical data using averaging or other methods which help us to keep database size reasonable.

                                              • Re: Replicated NetFlow
                                                RichardLetts

                                                Outage/downtime tracking reports -> if the switch that was down is removed from the database we lose that information

                                                Historical trending of interface data -> replacing a switch with a different model doesn't generally work because the interfaceindexes all change and there is no [builtin] tool to remap the old interfaceID/portID to the new interfaceID/portID.

                                                If you're doing any analysis of outages by switch type / software version then you've lost that information

                                                [see the idea for a reporting data warehouse with the aim of solving these kinds of issues]

                                              • Re: Replicated NetFlow
                                                banditone

                                                We use a un-supported DB setup that does not allow deletion of historical data so we do not lose it upon node deletion or re-naming, etc.

                                          • Re: Replicated NetFlow
                                            mrkuwait

                                            We are pushing netflow to Arbor and Arbor is pushing to our Solarwinds with no issues.

                                            shoot me an email if you need any help setting it up, cheers