This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

NetFlow detail records appear to contain duplicates

FormerMember
FormerMember

NetFlow detail records retrieved using PowerShell and Orion SDK appear to contain duplicates.

Granularity of Orion.Netflow.Flows.ObservationTimestamp appears to be minutes not seconds, and this appears to be creating duplicate records.

If granularity of Orion.Netflow.Flows.ObservationTimestamp is not the issue then why are we retrieving duplicate records.

...

SELECT

f.ObservationTimestamp

, ...

FROM

Orion.Netflow.Flows f

WHERE

f.ObservationTimestamp >= @BeginDate

AND

f.ObservationTimestamp < @EndDate

ORDER BY

f.ObservationTimestamp ASC

...

  $rs |

  select @{n='ObservationTimestamp';e={$_.ObservationTimestamp.ToString("yyyy-MM-dd HH:mm:ss.fff")}},ot,

... |

  Export-CSV -NoTypeInformation $AttachmentPath

P.S.

Beware: apparently SolarWinds support cannot or will not answer such questions. Not sure why we are paying for support!

  • You are correct that the granularity of ObservationTimestamp is minutes. The normal behavior of NTA is to coalesce flows to a 1 minute level before storage. If during a single minute the router reports two or more flows with the same 5-tuple (source IP, destination IP, source port, destination port, and protocol), NTA will combine these into a single record by summing their in/out bytes and in/out packets. In most networks, this greatly reduces the overhead of storing, indexing, and searching the flow data with little to no impact to the value of the data for network management.

    I would not expect you to see multiple records in the same minute from the same netflow source with the same 5-tuple. Is that what you are seeing?

    Regarding support for SWQL queries/Orion SDK issues - I monitor the Orion SDK forum and can work with you there to address any issues. Our phone support staff is not equipped to support the kinds of questions that come up in custom development, but I am. And I can engage the specific product development teams when necessary.

    Tim Danner

    Director of Architecture

  • FormerMember
    0 FormerMember in reply to tdanner

    Thanks for replying.  I do not expect support staff to answer questions but I do expect them to escalate to someone who can.

    Yes, we are seeing duplicate records within same minute and this is why we asked if NTA was storing records by seconds or minutes, or if we did something wrong using SDK.

  • If you have "ip flow ingress" and "ip flow egress" both configured on the same interface, occurring on more than one interface on a device, it is possible that you are getting double results that way. Not sure what else could cause it.

    (example, just in case my explanation was crappy)

    router1#

    int fa0/0

    ip flow ingress

    ip flow egress

    !

    int fa0/1

    ip flow ingress

    ip flow egress

    - If traffic flow is in from fa0/0 to fa0/1 out, conversations would appear twice in the log (reported once as ingress from fa0/0, and once as egress on fa0/1).

    - Corrective action is to remove ingress and egress from one of the interfaces completely.

    As an added stipulation, this makes an even bigger impact when a WAN optimizer is being utilized.

    What are the right NetFlow settings? Ingress vs. Egress | Paessler Knowledgebase