cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 10

NetFlow detail records appear to contain duplicates

NetFlow detail records retrieved using PowerShell and Orion SDK appear to contain duplicates.

Granularity of Orion.Netflow.Flows.ObservationTimestamp appears to be minutes not seconds, and this appears to be creating duplicate records.

If granularity of Orion.Netflow.Flows.ObservationTimestamp is not the issue then why are we retrieving duplicate records.

...

SELECT

f.ObservationTimestamp

, ...

FROM

Orion.Netflow.Flows f

WHERE

f.ObservationTimestamp >= @BeginDate

AND

f.ObservationTimestamp < @EndDate

ORDER BY

f.ObservationTimestamp ASC

...

  $rs |

  select @{n='ObservationTimestamp';e={$_.ObservationTimestamp.ToString("yyyy-MM-dd HH:mm:ss.fff")}},ot,

... |

  Export-CSV -NoTypeInformation $AttachmentPath

P.S.

Beware: apparently SolarWinds support cannot or will not answer such questions. Not sure why we are paying for support!

Labels (2)
Tags (1)
0 Kudos
3 Replies
Level 19

You are correct that the granularity of ObservationTimestamp is minutes. The normal behavior of NTA is to coalesce flows to a 1 minute level before storage. If during a single minute the router reports two or more flows with the same 5-tuple (source IP, destination IP, source port, destination port, and protocol), NTA will combine these into a single record by summing their in/out bytes and in/out packets. In most networks, this greatly reduces the overhead of storing, indexing, and searching the flow data with little to no impact to the value of the data for network management.

I would not expect you to see multiple records in the same minute from the same netflow source with the same 5-tuple. Is that what you are seeing?

Regarding support for SWQL queries/Orion SDK issues - I monitor the Orion SDK forum and can work with you there to address any issues. Our phone support staff is not equipped to support the kinds of questions that come up in custom development, but I am. And I can engage the specific product development teams when necessary.

Tim Danner

Director of Architecture

Thanks for replying.  I do not expect support staff to answer questions but I do expect them to escalate to someone who can.

Yes, we are seeing duplicate records within same minute and this is why we asked if NTA was storing records by seconds or minutes, or if we did something wrong using SDK.

0 Kudos

If you have "ip flow ingress" and "ip flow egress" both configured on the same interface, occurring on more than one interface on a device, it is possible that you are getting double results that way. Not sure what else could cause it.

(example, just in case my explanation was crappy)

router1#

int fa0/0

ip flow ingress

ip flow egress

!

int fa0/1

ip flow ingress

ip flow egress

- If traffic flow is in from fa0/0 to fa0/1 out, conversations would appear twice in the log (reported once as ingress from fa0/0, and once as egress on fa0/1).

- Corrective action is to remove ingress and egress from one of the interfaces completely.

As an added stipulation, this makes an even bigger impact when a WAN optimizer is being utilized.

What are the right NetFlow settings? Ingress vs. Egress | Paessler Knowledgebase

0 Kudos