Netflow compression/roll up

josh_d

Hi,

I understand how Orion compresses the Netflow records by 15 minutes intervals - NeflowSummary1 , 1 hour intervals - NeflowSummary2, and 24 hours intervals - NeflowSummary3 (for the sake of others I added a more complete description of the compression below).

 My question is - are the roll ups done on the 15,hour, and day - even though the record has a StartTime (which is just the starttime of the first record for that interval) of 01:24 it represents the Netflow for that unique set (dest, sources,tos,etc..) for the time span of 1:00- 01:59 (representing the 1 oclock hour). To clarify my question with a sample of (mocked up) data, the bolded record below taken from the NetflowSummary3 table would represent the netflow data - for that unique set (dest, sources,tos,etc..) - for the interval/time span of 8/3/2012 00:00 - 8/3/2012 23:59.Just to recap the question, are the roll ups done by the 15 (00:00,00:15,00:30,00:45), Hour (1:00,2:00,14:00,etc), and day or this this dependent on another factor (e.g. configuration, when the compression job runs,etc..)?

 

StartTime	NodeID	SourceIPSort	SourcePort	DestIPSort	DestPort	InterfaceIDRx	InterfaceIDTx	Protocol	ToS
7/26/2012 0:00	1	1000000000	0	2000000000	999	3	0	5	4
7/27/2012 0:00	1	1000000000	0	2000000000	999	3	0	5	4
8/3/2012 21:51	1	1000000000	0	2000000000	999	3	0	5	4
8/4/2012 0:00	1	1000000000	0	2000000000	999	3	0	5	4
8/5/2012 18:44	1	1000000000	0	2000000000	999	3	0	5	4
8/6/2012 0:00	1	1000000000	0	2000000000	999	3	0	5	4
8/7/2012 0:00	1	1000000000	0	2000000000	999	3	0	5	4

 

 

 Thank you in advance,

Josh

 

---------------------------------------------------------------------------------

1. We keep as-received data for the setting of “Uncompressed data”

2. We roll up as-received data form 1 min segments to 15 minute segments each 15 minutes and put it to NetFlowSummary1 table

3. We then roll up 15 minute segments every X hours to hourly data

NetFlowSummary1 - This table holds the summarized historical data for the first collapse level. The data are collapsed and moved to the NetFlowSummary2 table after certain number of hours. The data in this table summarizes a 24 hours traffic by default. (CollapseTrigger2InHours option in NetFlowGlobalSettings = 24)

NetFlowSummary2 - This table holds the summarized historical data for the second collapse level. The data are collapsed and moved to the NetFlowSummary3 table after certain number of days. The data in this table summarizes a 3 days traffic by default. (CollapseTrigger3InDays option in NetFlowGlobalSettings = 3)

NetFlowSummary3 - This table holds the summarized historical data for the third collapse level. The data are deleted after certain number of days. The data in this table summarizes a 30 days traffic by default. (RetainCompressedDataInDays option in NetFlowGlobalSettings = 30)

ondrej.salplachta

Hi,
your understanding is correct and here is explanation how we collapse data from time point of view:

The service periodically checks if conditions for collapsing are met (if elapsed required time since last collapsing and if there are any data to collapse), if so the collapsing begins. So the time when collapsing is executed depends on more factors - When service starts, when last collapsing was performed and when we have enough data to collapse (e.g. 1 hour of data to collapse from S1 to S2).

Then collapsing is performed in following way:
We take one interval (for 15 minutes it's e.g. 00:00-00:15, 0:15-00:30,... for 1 hour it's 00:00-01:00, 01:00-02:00,... and for days it's always since midnight to midnight). And we collapse all the same data into one record where StartTime then equals to minimum StartTime in the group.

Simplified example where we consider all other records are the same: 

Following data are in Detail table:
Time    Bytes
15:05  100 
15:08  50
15:17  300
15:29: 10

Here we have two groups (for collapsing into Summary1 where we have 15 minutes granularity):
15:00 - 15:15
15:15 - 15:30

So we collapse them into Summary1 table:
15:05  150
15:17  310

And the same is it with hourly or daily record, so to your example with bold record. In that case it was record with minimal StartTime in the group for whole day.

Regards,
Ondrej