3 Replies Latest reply on Jun 16, 2015 3:53 PM by dhanson

    NetFlow detail records appear to contain duplicates

    adeese2

      NetFlow detail records retrieved using PowerShell and Orion SDK appear to contain duplicates.

      Granularity of Orion.Netflow.Flows.ObservationTimestamp appears to be minutes not seconds, and this appears to be creating duplicate records.

      If granularity of Orion.Netflow.Flows.ObservationTimestamp is not the issue then why are we retrieving duplicate records.

      ...

      SELECT

      f.ObservationTimestamp

      , ...

      FROM

      Orion.Netflow.Flows f

      WHERE

      f.ObservationTimestamp >= @BeginDate

      AND

      f.ObservationTimestamp < @EndDate

      ORDER BY

      f.ObservationTimestamp ASC

      ...

        $rs |

        select @{n='ObservationTimestamp';e={$_.ObservationTimestamp.ToString("yyyy-MM-dd HH:mm:ss.fff")}},ot,

      ... |

        Export-CSV -NoTypeInformation $AttachmentPath

       

      P.S.

      Beware: apparently SolarWinds support cannot or will not answer such questions. Not sure why we are paying for support!

        • Re: NetFlow detail records appear to contain duplicates
          tdanner

          You are correct that the granularity of ObservationTimestamp is minutes. The normal behavior of NTA is to coalesce flows to a 1 minute level before storage. If during a single minute the router reports two or more flows with the same 5-tuple (source IP, destination IP, source port, destination port, and protocol), NTA will combine these into a single record by summing their in/out bytes and in/out packets. In most networks, this greatly reduces the overhead of storing, indexing, and searching the flow data with little to no impact to the value of the data for network management.

           

          I would not expect you to see multiple records in the same minute from the same netflow source with the same 5-tuple. Is that what you are seeing?

           

          Regarding support for SWQL queries/Orion SDK issues - I monitor the Orion SDK forum and can work with you there to address any issues. Our phone support staff is not equipped to support the kinds of questions that come up in custom development, but I am. And I can engage the specific product development teams when necessary.

           

          Tim Danner

          Director of Architecture

            • Re: NetFlow detail records appear to contain duplicates
              adeese2

              Thanks for replying.  I do not expect support staff to answer questions but I do expect them to escalate to someone who can.

               

              Yes, we are seeing duplicate records within same minute and this is why we asked if NTA was storing records by seconds or minutes, or if we did something wrong using SDK.

                • Re: NetFlow detail records appear to contain duplicates
                  dhanson

                  If you have "ip flow ingress" and "ip flow egress" both configured on the same interface, occurring on more than one interface on a device, it is possible that you are getting double results that way. Not sure what else could cause it.

                   

                  (example, just in case my explanation was crappy)

                  router1#

                  int fa0/0

                  ip flow ingress

                  ip flow egress

                  !

                  int fa0/1

                  ip flow ingress

                  ip flow egress

                   

                  - If traffic flow is in from fa0/0 to fa0/1 out, conversations would appear twice in the log (reported once as ingress from fa0/0, and once as egress on fa0/1).

                  - Corrective action is to remove ingress and egress from one of the interfaces completely.

                   

                  As an added stipulation, this makes an even bigger impact when a WAN optimizer is being utilized.

                  What are the right NetFlow settings? Ingress vs. Egress | Paessler Knowledgebase