13 Replies Latest reply on Mar 12, 2010 1:57 PM by chris.lapoint

    Remote Collectors for NTA

    mhh351

      I would like to know/see if NTA can be configured so that it receives information from REMOTE Collectors that forward to the central site.

      Other Vendors have this ability with the reasoning that collected data can go to the central site in blocks of data rather than stream the application server to death. They also provide for a SEPARATE database to handle the high amount of data and traffic.

      Since Solarwinds does this for NCM, NPM and the poller for VoIP is separate, could thought be given to have a separate database for Netflow.

      As others have mentioned in various threads on this site, with NTA running, SQL basically gets buried by the huge amount of incoming data. That is really not a good design to make NPM wait for another process.

       

      Thanks for the help and consideration.

        • Re: Remote Collectors for NTA
          Andy McBride

          Hi mhh351,

           

          We are investigating ways to off-load some of the work we are asking SQL to do. It is a priority for us.

           

          Andy

            • Re: Remote Collectors for NTA
              mhh351

              If I may be so bold, may I suggest a stripped down version of the NTA function be a no cost license. It would not be usable with any other software. It's only function would be to collect and send flow information.

              Have this collector software set up in lesser devices, such as good but old PCs or old but sufficient servers placed in various parts of the network. This would be in a "probe" sort of style. It also gets you ready, at Solarwinds, for what I think you are visualizing!

              Add the function that NTA will look for these devices based on criteria, either supplied by IP address and name or have the collector "call home" to be recognized by the NTA program "Job". Your criteria should be proprietary to the point that you select a TCP Port as well.

              Another thought would be for the collectors, no matter whether they are remote or standard, to send to the central site SQL server on their own.

              I still vote for a separate database and maybe even database server as a requisite set up. Stay away from NPM at all costs. This sort of traffic and the NPM traffic will not coexist well due to the constant chatter or each kind of traffic.

              These collectors then dump the data at specified times (2 to 5 minutes) directly to the database. The NTA program can then get the information directly from the DB without further processing. The only thing that NTA would have to do, is recognize new flows and/or devices for its functions and displays.

              The initial recognition would be somewhat slow so as to line up with the Node ID in the NPM database (NCM, too) and getting it applied to the NPM Web Page for that device.

              I hope that gets you going and helps with the thought process.

                • Re: Remote Collectors for NTA
                  mhh351

                  Any Thoughts?

                  • Re: Remote Collectors for NTA
                    pserwe

                    I'll second the motion, although I'll go a step further..  Netflow should dump data into it's OWN SQL database.  No reason the website can't pull data from multiple sources.  Hell, it does already when we really don't want it to, like NCM.


                    Peter

                      • Re: Remote Collectors for NTA
                        Donald_Francis

                        We run netflow with about 250 interfaces and we slam the heck out of our SQL box and I suspect it is due to netflow.

                        I would be very handy if it was a seperate DB so you could place it on another SQL box or at least on a diff set of drives.

                        It would also be great if instead of their being a DB update for every flow if they were cached for some short period of time and wirtten to the DB in blulk.

                          • Re: Remote Collectors for NTA
                            Andy McBride

                            Hi Donald,

                            We are investigating ways to off load some work from SQL. No Dates to promise

                             

                            Andy

                            • Re: Remote Collectors for NTA
                              Andy McBride

                              I hear you guys. These are all options we are considering.

                              Andy

                                • Re: Remote Collectors for NTA
                                  jeff.stewart

                                  I do like the thought of these ideas, but additional software and hardware cost come in to play.  I will say that v3.5 has made huge gains in performance than other version.  If the performance keeps increasing with each version I see no need for the ideas mentioned. 

                                    • Re: Remote Collectors for NTA
                                      mhh351

                                      Just a kindly comment. When we run a Worldwide network, having it take all of this Netflow data across the WAN links becomes costly froma data standpoint. Consolidating that data and sending it in compressed or pre-processed form is much better. OTher vendors do that and the efficiency is much greater. Links that are already busy and costly should be used as efficiently as possible. If this was just for domestic sites, we would not care as much.

                                      As for the amount of data going to ONE database, that is just not a good design. Each Database should have its own purpose if not its own server. If one looks at the amouont of data transactions that NPM does already to the DB server, one would not want to add that kind of traffic or amount of data to the DB from Netflow types of transactions.

                                      Response time is very important to improve upon and your point is a good one. When we try to scale the overall application/database relationship, we want to make sure that other applications such as NPM do not have to suffer for either the size of the database or for the other polling/gathering that is going on.

                                      Thanks for the discussion!

                                        • Re: Remote Collectors for NTA
                                          chris.lapoint

                                          Thanks for the continued comments and discussion.  I can tell you that scalability and simplified visualization of data are absolutely key themes for our upcoming releases.   Our goal is to take an evolutionary approach to address the use-cases described above.  As Jeff noted above, we made some significant strides in the 3.5  release and our upcoming 3.6 release will continue to improve scalability and performance.

                                          For those continuing the discussion, it is extremely helpful if you can list out your use-cases (i.e. what specific problems are you trying to solve) as mhh351 did above in the first paragraph.   This way we can prioritize and ensure that as we look at ways to evolve NTA architecture for the big win (e.g. separate database, standalone remote collector, etc.), we can also knock off some smaller wins along the way for you.

                                          Keep the great feedback coming!

                                          -Chris

                            • Re: Remote Collectors for NTA
                              kdouglas1

                              Any update on this?

                              We're in a competitive situation trying to decide on the direction for our Enterprise wide NMS.  While Orion wins hands down for usability, my past experience with NTA (a key component of our solution) has been less than stellar when trying to collect information from a fair number of interface (collecting from six GigE interface on a beefy box allowed Orion to perform well, but in my new environment I'm looking at about 100 interfaces). 

                              The competitor supports separate NetFlow collectors that don't impact the performance of the core application.  They can support two or three separate collectors that optimize the date before sending it to the core application. 

                              I'm concerned about recommending Orion given the performance issues I've seen.  Off load the collector function AND add NTA collector function to the Scalability Engines.  

                              Ken