18 Replies Latest reply on Aug 19, 2009 2:45 PM by chris.lapoint

    netflow v3 stops resolving dns names

    gwsample

      I am having an issue with netflow resolving dns names.  This started after upgrading netflow from v2 to v3.  I am running Orion v8.5.1 w/ SP3 and Netflow v3.  All my nodes in Orion show up in netflow with the correct name, reverse lookups work fine.  About a month after the upgrade, all the names disappeared and netflow only showed IP addresses.  Also, the nslookup button on the endpoint details screen would do nothing.  I verified reverse lookups work fine on my orion server, as far as windows is concerned.  Restarting netflow and rebooting the server had no affect.... So I opened a support ticket... thier answer: 

      "If you truncate the FlowCorrelation table within the NetperfMon database, all netflow endpoints should be resolved to their DNS names again."

      I did this, and after wating 2 days for NTA to automatically lookup hostnames, everything was back to normal.  I expressed concern with Solarwinds support this was not an acceptable fix.  Truncating this table everytime netflow looses it mind, is not a fix.  Well... it has been about a month, and gues what, netflow has stopped resolving dns names again.... I can fix this by truncating the table again, but would like to find a REAL solution.  Anyone have this same problem, or have a fix for it?

      Thanks,

      Gary Sample

        • Re: netflow v3 stops resolving dns names
          andrew

          I am having the exact same issue.  Did you ever find a solution?

          How do you "truncate the FlowCorrelation table within the NetperfMon database".  Are you talking about just removing all the records from that table?  Is that going to cause any data loss?

            • Re: netflow v3 stops resolving dns names
              Andy McBride

              Hi Guys,


              This is a known bug in 3.0 and we are addressing it. The flow correlation table will rebuild so no history lost.


               Andy

                • Re: netflow v3 stops resolving dns names
                  andrew

                  So, that means it is safe to run "TRUNCATE TABLE FlowCorrelation"?

                    • Re: netflow v3 stops resolving dns names
                      davidmaltby

                      It is safe in that the FlowCorrelation table only keeps a resolution of DNS resolutions.  Once you delete it, of course, the IP addresses will not have a way to resolve in your graphs, until new NetFlows come in with those IP addresses in them.  Then the service will see that they are not in the FlowCorrelation table and will make new DNS queries.  The history will be retained, but if no new NetFlows come in for a particular IP address, then since the graphs join on the FlowCorrelation table, I believe that they won't show up until a new DNS query is made on them.  Let me come up with a better way to do this instead of truncating the table.  I should have some T-SQL  for you to run instead this evening that will do the same thing in effect, but doesn't have this type of side-effect.


                      Thanks,

                        • Re: netflow v3 stops resolving dns names
                          gwsample

                          yes, I am still having this issue as well.  I either have to truncate the table, or re-applying the latest netflow service pack and rebooting seems to fix it temporarily.

                          • Re: netflow v3 stops resolving dns names
                            davidmaltby

                            Here is what I really suggest that you run on the NetPerfMon database instead of truncating the FlowCorrelation table..


                            DECLARE @CacheExpiration datetime
                            SELECT @CacheExpiration = dateadd(second, 1, LastCacheExpirationCheck) FROM NetFlowCorrelationState
                            UPDATE FlowCorrelation SET CacheExpiration = @CacheExpiration


                            This will cause all the DNS entries for all IP addresses to expire and the service then will start performing DNS queries against all of them.  Keep in mind that this is not a trivial task that you are forcing the service to do.  It WILL take some time to make a network called to resolve each of the IPs in the FlowCorrelation.  You can check on the service's progress by periodically running the following T-SQL:



                            SELECT Count(*) FROM NetFlowAddressToResolve


                            It will give you a count of how many more DNS queries that the service needs to make before it is finished.  Note: If you have a bunch of sources sending NetFlows at the same time, this process can take a long time.


                            Hope this helps!  Let me know!


                            Thanks,


                              • Re: netflow v3 stops resolving dns names
                                andrew
                                I don't think this worked.  I ran the first query and it updated about 7 million rows.  Then I ran your second query and it says 0.  I tried restarting the NetFlow service, but that didn't make any difference.  I then ran:
                                SELECT count(*) FROM FlowCorrelation WHERE Hostname = '';
                                and that returned "7026560".
                                I have waited a few hours and at most I have seen NetFlowAddressToResolve go to 2.  The number of entries with no hostname in FlowCorrelation keeps going up.  If I click the Lookup button on the website, it will resolve the name properly.  But, I can't do that for 7 million records!
                                  • Re: netflow v3 stops resolving dns names
                                    davidmaltby

                                    So the values must be messed in somewhere for these dates.  Can you look in the FlowCorrelation table and look now at what the dates are for the CacheExpiration field?  (All 7 million should be the same since we ran the query)


                                    Furthermore if you open the NetFlowCorrelationState table, what value is in there? 


                                    What is the computer system date/time on both the Orion/NTA machine and the SQL database server?  Are they in the same timezones?


                                    Thanks,