cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 9

How Do You Make The Most Of Your Flow Data?

As a long-time exporter of flow data from edge devices, I've run into the issue of data underuse. I usually export from edge devices: WAN routers & firewalls. I've not personally found too much use in exporting flow data from core devices, as that's rarely where I need to do flow analysis. The edge devices have been the key in my experience, answering questions such as:

  • Who's using all the bandwidth?
  • How long has a given flow been alive?
  • What's the traffic mix look like across this link?

What occurs to me is that I usually only look at flow data when I've got a problem, most often when a link is saturated and I need to quickly nail down the culprit. That said, I know that's a waste of interesting historical data, if only I'd take the time to look at it. There's just so much of it to plow through, that I usually don't take the time. It seems like there's always so many more pressing things to do that I don't take the time to make large chunks of historical flow data digestible.

To kick off a discussion, I'm curious to hear the answer to a few questions.

  1. Other than troubleshooting, how do you make use of your flow data?
  2. How often do you look at flow data summaries, and in what form (on-demand via HTML form, automated reporting, CLI top-talkers, etc.)?
  3. Summaries have the side effect of masking more granular data, i.e. smaller flows that might be interesting. In your view, is this a concern, and if so, how do you work around it?
  4. How old does flow data have to get before it's no longer useful? For the sake of SQL, I was only keeping 7 days worth of flow data in NTA, assuming that it would be very unlikely I'd need to go back further than that. That was true most of the time, but there were times I wished I could dig back further.
Labels (1)
29 Replies
Level 9

While we don't really collect too much flow data, we also don't often look at what we do collect. For the most part- only when we notice a problem. I agree that it's hard to setup a system to summarize this data without losing some important details. I guess that's kind of the nature of the beast. You can either look at all the granular data, or you can miss some of the granular data... lol. I rarely need to look back over a week for anything.

0 Kudos

Other than troubleshooting, how do you make use of your flow data?

Answering the daily question....Why is _______ slow?

How often do you look at flow data summaries, and in what form (on-demand via HTML form, automated reporting, CLI top-talkers, etc.)?

At least once a day.

Summaries have the side effect of masking more granular data, i.e. smaller flows that might be interesting. In your view, is this a concern, and if so, how do you work around it?

We dig into the small streams as required by the IDS.

How old does flow data have to get before it's no longer useful? For the sake of SQL, I was only keeping 7 days worth of flow data in NTA, assuming that it would be very unlikely I'd need to go back further than that. That was true most of the time, but there were times I wished I could dig back further.

I am getting ready to install NTA 4.  I am looking forward to the minute by minute data.

Level 10

Other than troubleshooting, how do you make use of your flow data?

  • For QoS validation.  In cases where companies outsource QoS deployment to the ISPs, NetFlow is great to validate whether the service provider is classifying your traffic per contract.

How often do you look at flow data summaries, and in what form (on-demand via HTML form, automated reporting, CLI top-talkers, etc.)?

  • Twice a day.  on-demand via HTML

Summaries have the side effect of masking more granular data, i.e. smaller flows that might be interesting. In your view, is this a concern, and if so, how do you work around it?

  • For me, not a concern as I am leveraging performance data via NPM

How old does flow data have to get before it's no longer useful? For the sake of SQL, I was only keeping 7 days worth of flow data in NTA, assuming that it would be very unlikely I'd need to go back further than that. That was true most of the time, but there were times I wished I could dig back further.

  • Depends.  A week is usually fine, but if I had the capability to do forecasting/capacity planning, then up to 13 months (summarized of course) data would be nice.
0 Kudos

Other than troubleshooting, how do you make use of your flow data?

Pinpoint traffic issues.  Most issues are self inflicted by users.

How often do you look at flow data summaries, and in what form (on-demand via HTML form, automated reporting, CLI top-talkers, etc.)?

The Top xx Netflow Sources by % Utilization is always up and on the front page of Orion.

Summaries have the side effect of masking more granular data, i.e. smaller flows that might be interesting. In your view, is this a concern, and if so, how do you work around it?

As alerts come in we use NF to do the initial drill down.   Flow Navigator is great!

How old does flow data have to get before it's no longer useful? For the sake of SQL, I was only keeping 7 days worth of flow data in NTA, assuming that it would be very unlikely I'd need to go back further than that. That was true most of the time, but there were times I wished I could dig back further.


A week of data is fine.

RT

0 Kudos

  • I built a cost allocation model identifying servers owned by business lines, export conversation data and calculated the associated consumption of network resources..
  • I look at the flow data to validate that new applications are connecting via the routes I expect them to and that they fail properly to backup routes.
  • I look at the characteristics of various applications and advise business owners how to enhance them to improve efficiency and reduce their costs.

What I need :

  • IP address management linkage to NPM and a much improved interface / linkage / import mechanism
  • The ability to report on netflow conditions ... Traffic rate is non-existent between servers ... traffic rate is less than / greater than same time last week etc.
0 Kudos
Level 13

For the most part my flow data has gone unused.  It is a solution in search of a problem.

On occasion it has been useful to discover the largest traffic types going through a particular link.

0 Kudos
Level 21

We have some in-house built software that we use for flow data, I personally don't find it very useful.  I am curious though, do you always have the flow collections on or do you only turn it on when you want to track down an issue?

0 Kudos

Hi Ethan,

The most common use case I come across for Flow is WAN analysis. One of the main drivers for that is the fact that flow options exist on most routers. From a storage point of view I think most people want at least 1 weeks’ worth of data which is not aged in any way.

Flow does have its problems as its mostly focused at layer 3 and 4 so it can be difficult to monitor applications which use common ports like TCP 80 or applications which port hop. NBAR is seen as a solution here but it only available on a small number of devices, we had so much demand for this sort of functionality that we developed a feature for our LANGuardian product called CBAR which takes its feed from a regular SPAN or mirror port.

Another common request I see when it comes to flow monitoring is the ability to map flows to usernames. This makes it more 'management' friendly as you can see who is using up bandwidth rather than showing IP addresses or hostnames. Does anyone find this type of feature useful?

Darragh

0 Kudos
Level 9

Honestly I'm in about the same boat as you, Ethan. Mostly I only look at this data when I have to troubleshoot a problem. Occasionally I'll look at it when there isn't a problem but just to see how many people are streaming Pandora, are on FaceBook, etc. That gives me a good idea of what I can normally expect to see which makes spotting problems later easier.

0 Kudos
Level 12


Every day as part of daily activity, I look at NetFlow Sources for all devices (Cisco only) to make sure they have current date and time.  This helps us to know NTA is working correctly.

0 Kudos
MVP
MVP

I use it for WAN links to monitor information going back and forth from Server to computer and vice versa. It is especially useful when users or even Server administrators are complaining that circuits are slow. You would be surprised at what users are doing that slow down the WAN circuits.

0 Kudos
Level 10

WAN tunnel links is really the main use. This is an area where we really need to expand the proactive monitoring/alerting to catch a utilization issue before the site is calling and telling us.

0 Kudos

We use NFSEN as a lightweight tool to notify us when we have DDOS attacks, showing xx amount of traffic/%.  That information we keep until 7.9GB storage limit is reached.  This is only to have a realtime alert for events as they happen.

We use NTA to show us all the details for the IP we find via NFSEN.  I believe we are keeping 30 days of NTA (3.11) data.

I would love to use NTA more, but it is a hard sell to mgmt when we have a different (NFSEN) server that cost $0 and delivers on what matters most.

NTA is used more of a, "fill in the gaps with this info" server.  It is much slower (albeit v3.11) and apparently not as smooth to navigate around.

0 Kudos
Level 9

  • Other than troubleshooting, how do you make use of your flow data?
    • My group is in the middle of an enterprise wide network refresh so we're using flow data for both troubleshooting and for capacity planning. It has proven to be invaluable in that capacity when we are trying to justify upgrading WAN circuits
  • How often do you look at flow data summaries, and in what form (on-demand via HTML form, automated reporting, CLI top-talkers, etc.)?
    • We look at flow summaries at least twice a day. It helps us be a bit more proactive with issues on the WAN. It also ensures we can answer management's questions when they ask instead of having to look up the answer first.
  • Summaries have the side effect of masking more granular data, i.e. smaller flows that might be interesting. In your view, is this a concern, and if so, how do you work around it?
    • In capacity planning the summaries are perfect. For troubleshooting the summaries get us pointed in the right direction as to where we need to look. Granular data hasn't been a real concern yet.
  • How old does flow data have to get before it's no longer useful? For the sake of SQL, I was only keeping 7 days worth of flow data in NTA, assuming that it would be very unlikely I'd need to go back further than that. That was true most of the time, but there were times I wished I could dig back further.
    • We are keeping 14 days worth of flows at any given point in time. Usually a report is generated every 2 weeks to ensure we have history to show to management when we ask them to approve something.
0 Kudos
Level 12

  1. I'd want to do much more but right now it's just troubleshooting really...
  2. Only when a problem arises
  3. We can live with less precision as long as we cover the "big" stuff so it's not an issue for us
  4. We currently don't keep anything and analyze live flow, but we are planing on adding that functionnality to our network next year and I believe 7 days would be a good start.
0 Kudos
Level 10

I use it for real-time analysis of possible attacks.  Being able to quickly see the 1 or 2 IP addresses bombarding traffic means it's all the more quicker to banhammer those IPs.

0 Kudos
Level 15

  1. Other than troubleshooting, how do you make use of your flow data?
  2. How often do you look at flow data summaries, and in what form (on-demand via HTML form, automated reporting, CLI top-talkers, etc.)?
  3. Summaries have the side effect of masking more granular data, i.e. smaller flows that might be interesting. In your view, is this a concern, and if so, how do you work around it?
  4. How old does flow data have to get before it's no longer useful? For the sake of SQL, I was only keeping 7 days worth of flow data in NTA, assuming that it would be very unlikely I'd need to go back further than that. That was true most of the time, but there were times I wished I could dig back further.

1) I like to setup reports for managers who have responsibilities around the scalability of the network, as well as network, videoconf, and VoIP teams.

2) Tier I/II teams will usually use this tool a LOT to combat the "my network is slow" tickets

3) If there is a specific problem I am looking at, I will try to get more granular, otherwise I blindly trust my NTA Summary overlords.

4) I keep the 30 day standard most others here have mentioned, depending on the capabilities of the SQL box.

0 Kudos
Level 11

I had NTA monitoring traffic coming in across WAN links, mostly the information was used for troubleshooting purposes.  I don't have any issues with the summaries masking information, but at times it was the "unclassified data" that I was looking for.  All of the NTA data was kept for 60 days.

I no longer use NTA not because the data isn't valuable, but because the licensing for NTA has to match that of NPM.  We recently had to increase the license level of NPM and it didn't make sense to spend thousands to upgrade NTA to 250 nodes when we're exporting data from 5 sources.

0 Kudos
Level 9

Depends on the client, small office, small uses only monitor a few bits.

But the larger clients with 1000s, we mainly monitor nodes response time, interfaces by traffic and % util. Same, just 7 days worth.grantallenby

0 Kudos
Level 13

we do two main types of flows

! client side initiated traffic ( as seen from the  access switches)

- looking at east west traffic

- as well as north bound traffic

and

1 Wan traffic coming in

- inet initiated coming in

- response the client responses

0 Kudos