Differences between Orion.Netflow.Flows and Orion.Netflow.FlowsBy*

Question

In Orion database there are seven tables that contain Netflow information. They have 36 fields in common, such as the SourceIP address, Number of bytes, SourceHostname etc. The difference is extra fields, for example ByHostname adds the fields ASID, CountryCode, Domain, Hostname, IP, IPGroupSegmentID, and PartnerIP. Some of the extra fields appear to be duplicates of the common fields. For example the extra field Hostname always (in my case and my limited sample) matches either the common fields SourceHostname or DestinationHostname. This also applies to the extra field IP and PartnerIP. Are the extra fields there as a programming convenience (for easier flow search) or for some other purpose that I haven't come across? I would like to use either Orion.Netflow.Flows.ByHostname or Orion.Netflow.Flows.ByIP as they capture more information, but they appear to report the same flow multiple times, just with different "key" values. I need to make sure I don't drop flows by accident.

Thanks

Martin.Krivanek · Accepted Answer

Hi, In essence Orion.Netflow.Flows contain all flows. FlowsBy* entities offer different view over that data. Their main purpose is to provide easy way to query directional fields (e.g. SourceIP and DestinationIP) regardless of their direction (as if there is IP field and you don't care whether it was received as source or destination). So yes, it's more for convenience and ease of use. The purpose can be demonstrated with an example: let's say there is only one flow in database. 
 SELECT SourceIP, DestinationIP, SUM(Bytes) AS Bytes FROM Orion.Netflow.Flows GROUP BY SourceIP, DestinationIP 
 This would return only one row: SourceIP DestinationIP Bytes 1.1.1.1 2.2.2.2 100 Now if you would use FlowsByIP entity as follows: 
 SELECT IP, SourceIP, DestinationIP, SUM(Bytes) AS Bytes FROM Orion.Netflow.FlowsByIP GROUP BY IP, SourceIP, DestinationIP 
 The result would be two rows: IP SourceIP DestinationIP Bytes 1.1.1.1 1.1.1.1 2.2.2.2 100 2.2.2.2 1.1.1.1 2.2.2.2 100 Note that it only appears as if database contain two rows (in reality there is only one). IP column is virtual and contains source/destination field for first/second row - this virtual duplication happens for each flow. This is very useful if you need to filter or group by IP address disregarding its original direction. Same applies to other fields, for example Domain field is also virtual and would hold value from either SourceDomain or DestinationDomain. If you don't need any of this logic, I suggest to use Flows entity. If you need to query directional fields disregarding their direction these entities are convenient way how to do that. I hope this helps, let me know if you have any further questions. Martin