Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Missing Chart Data in NetFlow v3.0

I would assume that much of the missing data has to do with the V3 performance problem not processing all the data in the queues, but maybe not so I am asking... I was tracking down some performance issues today with Orion and while the router serial interface was at ~100% since noond today, none of the flows causing the problem were showing up in NetFlow... Will they ever?

From what I can see there is a great deal of traffic not being processed due to what I assume is the performance problem with the SQL database and it getting behind. With a fairly small SQL server and a single NetFlow enabled router with a fairly low number of flows we are seeing I/O queues >4000 on the SQL server storage drive.

I am not sure what percentage of the flows we are losing but it seems pretty large... I have screen shots and charts to indicate better what I am seeing but here might not be the right place...

Find more posts tagged with

Accepted answers

All comments

aLTeReGo

I have screen shots and charts to indicate better what I am seeing but here might not be the right place...

Here is definitely the right place. Show us what you're seeing.

CoreyMac

From the router IP flow cache you can see the partners in the traffic. These are not the ones listed in the NetFlow conversations for this 10.20.0.30 IP address. Also, the charts for the NetFlow traffic do not jive with the charts for the interface traffic, even though we know where it is coming from the NetFlow traffic is peaky and only shows ~5MB per 15 minutes as a max, but the Orion interface charts (and the router itself) show a pegged interface at 256Kbps (~22MB/15minutes)... Seems to me we are missing a large amount of data in the flow traffic… We are missing both data and hosts, so to me that constitutes flows we are dropping.

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP Pkts
Se0/2/0       69.26.185.67    Vl1           10.20.0.30      06 0050 1210    54
Se0/2/0       69.26.185.123   Vl1           10.20.0.30      06 0050 1229   629
Se0/2/0       69.26.185.123   Vl1           10.20.0.30      06 0050 1228   649
Se0/2/0       69.26.185.123   Vl1           10.20.0.30      06 0050 122B    14
Se0/2/0       69.26.185.123   Vl1           10.20.0.30      06 0050 122C    65

If the ratio of flows to packets corresponds to about 11-14 flows per packet as it seems, and we are not processing the ~325K packets left in the Raw Packet Queue when we restart the service every night at 4am, that amounts to 3.5-4.5 million flow stat updates we are missing every day.

Catoosa 256K NetFlow MB-15 min.JPG

CoreyMac

As you can see from this Orion chart of the interface traffic and with the router showing the flow stats, several things are amiss... The traffic according to the router flows is HTTP port 80, and the interface charts show that the traffic is continuously busy at near 100%. The NetFlow charts show a mix of traffic, much lower data rates, no remote hosts that are seen by the flows in the router, etc... Looks very much like missing flow data to me anyway...

Catoosa 256K Circuit MB-15 min.JPG

mcbridea

Hi CoreyMac,

I have had this tested here and we found that the total calculated traffic and the NTA traffic were almost the same. The total traffic was slightly higher than the NTA peaks which is what I would expect. Obviously this is not the case in what you are seeing.

NTA won't measure things such as NetFlow, Syslog and traps from the device but chances of this being a large % of your data is low. I suggest you go to Netflow Settings -> Applications and Service Ports -> Edit and choose :Monitor All.

Give this some time to collect and see if you now are closer to parity. If this does not do it then the old packet capture may shed some light. Either way I would like to know what you find.

Andy