SolarWinds Netflow Traffic Analyzer will keep detailed information for 60 minutes by default. This means you can get up to 1 minute granularity of flow information for the last 60 minutes. If you look for information from 2 hours ago, only the last hour will be represented up to 1 minute granularity. Below is a description of how exactly the NetFlow collector and database process information. Hopefully this will help in planning and understanding what you can do with the product to meet your needs.

1. Receive data and store in a temporary processing queue in memory

2. Every minute, this temporary queue will be processed before writing to the database. The main two processing steps are collapsing flows and Top Talker Optimization.

A. Flow Collapsing is the process of taking related flows (same source interface, source IP and port, destination IP and port) and aggregating the data into one record. When this information is written to disk, we will mark the time that the collapsed data was written. This means you cannot see granularity within this 1 minute interval.

For example: if you have this table of all collected raw flows

    

Interface

Source IP

Source Port

Destination IP

Destination Port

# Bytes

# Packets

Fa0/1

1.1.1.1

80

2.2.2.2

80

1024

2

Fa0/1

1.1.1.1

80

2.2.2.2

80

1024

2

Fa0/1

1.1.1.1

80

2.2.2.2

80

512

1

Fa0/1

1.1.1.1

443

2.2.2.2

443

1024

2

Fa0/1

1.1.1.1

443

2.2.2.2

443

1024

2

It will be collapsed into this table

    

Interface

Source IP

Source Port

Destination IP

Destination Port

# Bytes

# Packets

Fa0/1

1.1.1.1

80

2.2.2.2

0

2560

5

Fa0/1

1.1.1.1

443

2.2.2.2

0

2048

4

B. Top Talker optimization is the process of recording the flows that represent the most traffic in your network. By default, this value is set to 95%. For some users who need auditing precision, they change this setting to 100%. Based on research with internal testing and customers, 95% has the best results. Most of the other packets that are not being recorded usually have only sent a few packets and are not interesting from a traffic utilization perspective.

3. After the flows are collapsed and the top talkers are filtered, the data is written to disk. This data is written to a NetFlowDetail table. The exact table name depends on the node ID and a timestamp.

4. After the “Keep uncompressed data for X minutes” interval, this detail table will be further collapsed (compressed) and written to the NetFlowSummary1 table. The default of this interval is 60 minutes and can be increased up to 240 minutes in the web UI. The setting can be manually modified in the database by changing the NetFlowGlobalSettings.RetainUncompressedDataIn15MinuteIncrements value. We do not recommend you increase this value above what can be done through the web UI. The settings in the web UI are based on extensive testing with customers and internal performance testing. Once the data goes to this table from the Detail table, you can only see the traffic at a 15 minute granularity. That means, if your interval is from 10:00 – 10:15, you won’t be able to distinguish if the traffic was sent at 10:02 or 10:13 – we will just show you that is occurred in that range.

Here is a screenshot of the settings from the website with the settings.

image

5. 24 hours later, this information will be collapsed from 15 minute granularity to 1 hour granularity and stored in NetFlowSummary2 table. There is no way to modify this interval (24 hours) in the web UI. You can modify it manually in the database by changing the NetFlowGlobalSettings.CollapseTrigger2InHours value. This means, when you look at data from the previous day, you will only see that the traffic occurred in a 1 hour period.

6. After 3 days, this 1 hour data will be collapsed into daily data and stored in the NetFlowSummary3 table. This interval is set by the NetFlowGlobalSettings.CollapseTrigger3InDays value (default 3 days). This means, you will only know what day the traffic occurred and not what hour or minute. This data is kept for 30 days by default. This interval can be increased to up to 3650 directly in the web UI.

7. When flow data is expired (based on the above settings) the expired data will be permanently deleted from the database based on the “Delete expired flow data” interval (available in the web UI, once a day by default).

 

It's worth noting that this is just an example. All intervals are relative to the time when the flow arrives at the service. The various aggregation steps can happen at a later time if the service is too busy. However, the daily aggregation will always occur once a day.