Enable Deduplication for NTA

Enable deduplication for NTA. We have utilized tools in the past that had deduplication capability and it was very convenient for troubleshooting.

Find more posts tagged with

deduplication

ntadeduplication

Status: None

Comments

slebbon

I don't understand what the request is for? Deduplication of what data exactly? Could you elaborate more?

rschroeder

I'd appreciate several deduplication options:

It's best for me to see the traffic recorded once on its way across my network. But if I enable Netflow (ingress and egress) on my core routers and on my L3 Distribution switches, and on edge routers, and on remote WAN routers, and on remote L3 Distribution Switches, and on all L3 interfaces at the data centers, it's possible I'm seeing the same packet counted in and out eight times. It'd be cool if a Network Admin didn't have to THINK, and could just enable Netflow on all interfaces in both directions. On the other hand, we really need to know what we're doing, instead of blindly enabling ingress and egress on all L3 interfaces, no matter how much easier that is.
Simply identifying the duplication in a single interface's ingress and egress is useful. Clicking an NTA "deduplicate" button would be an easy way to filter out traffic that's all the same on one interface.
Expand that same idea for deduplication across and entire network's flow. If a packet crosses a dozen L3 interfaces to get to its destination, it might be useful to see that packet recorded a dozen times in NTA. Or it may skew the traffic statistics so they look ridiculously large. They could even look to be greater than the capacity of some WAN pipes the packet crosses, and that's not intuitive and it's not good to show that data to Management as a reason for problems, or as a reason to justify increasing WAN pipe size while increasing the cost of the WAN.
Provide a flag or icon on NTA graphs that indicate whether Deduplication is enabled or disabled. Maybe even make it more granular and show multiple icons to indicate if deduplication is partially enabled, or enabled on some but not all L3 interfaces.

When I see 24 TB/hour transmitted across my network for weeks on end, one has to wonder how much of that is duplicate information consolidated in the graphs by monitoring ingress and egress traffic across all L3 interfaces.

Is this useful information, or is it leading a person to make incorrect judgments because some traffic is recorded twice, or four times, or sixteen times as it crosses the multiple L3 data center, core, distribution, and WAN interfaces?
How differently would the graphs look if only ingress, or only egress, were being monitored?
Where's the guideline that tells us the best way to do one or the other?
Where's the radio button to filter out duplicate traffic from the graph?
Where's the flag that indicates duplication is present, or that deduplication has been applied to the graph?

RaviK

In simple terms, traffic entering a WAN interface and exiting a LAN interface will show up NetFlow traffic twice if Netflow is enabled on both interfaces. There may be use cases where we have multiple interfaces and we need to enable NetFlow (both Ingress and Egress) on all of them. Chances of duplicating the traffic is very high, hence de-dup is very much a necessity. Else your NetFlow stats including charts will show up the same traffic multiple times.

abdhijasharma

Yes, yes, a hundred times yes...!!

It is an ongoing steady headache with one of the customers, who wish to monitor each and every interface of each and every device, and it becomes tedious explaining everytime a new resource comes in, as to why the bandwidth doesn't look what it should, and explaining them how to understand the data in this situation is a whole 'nother monster... Quite frankly it is something that should have been thought about and implemented much earlier....

stevenstadel

We are currently using Riverbed to capture the traffic, perform the deduplication, and then forwarding it to Orion. Not an optimal solution.

rschroeder

Bump.