Request for guidance to reduce FPS in our SW environment

Hello Team,

I hope this message finds you well.

We have submitted a request to the SW support team for a system health check and to address the issue of slowness. Diagnostic logs have been provided for further analysis. Subsequently, we received below observations from the support team, which highlighted the need for us to address these issues to resolve the slowness. I am not familiar with these issues and would appreciate step-by-step guidance on how to resolve the FPS issue.

Attached is the snapshot provided by the software team for your reference. Could you please advise on how to reduce the FPS to rectify this problem, and also explain the meaning of 'instance limit'?

As per the SW team, there's a limit of 50 thousand fps, whereas you are receiving 5373413, which is extremely high and even higher than the total instance limit[300k].

Find more posts tagged with

support

Application

netflow

solarwinds

solarwinds netflow traffic analyzer

nta

netflow_traffic

Accepted answers

All comments

vinay.by

These are the guidelines that you are talking about

https://solarwindscore.my.site.com/SuccessCenter/s/article/What-is-the-maximum-amount-of-flows-per-second-that-can-be-processed-in-NTA?language=en_US

In simple terms you will need to balance out the load in your environment, I am assuming you are a monitoring tools admin and you don't work on operations or platform or infrastructure side, hence this cannot be done by yourself you will need to work with the folks who manage the end devices.

1. Balance out the load between the pollers you have - always remember 'you never overload your primary poller' (Main Poller) which in your case seems like its already been done, you are close to maxing out your Main Poller.

2. Always use Additional Pollers for polling most of your stuff and if your Additional Poller is maxed out that is when you use your Main Poller for polling (there are other reasons as well why we onboard devices on to main poller but lets stick to the basics), from the above screenshot it looks like you are killing your Main Poller.

3. Firstly, you have to move most of the devices out of your Main Poller onto Additional Poller - balance out the load and don't kill your Main Poller.

4. Before you do point 3, check with the device admins to which SolarWinds IP is netflow configured to, is it just your Main Poller IP or can it also send to your Addtional Poller IP, if its configured to your Main Poller and Additional Poller then its a easy task for you, if that is not the case then ask them what options do you have - can they send it to both your SolarWinds IP's or can they switch it to Additional Poller IP. If they can easily make the switch to your Additional Poller IP - then come up with a list of devices that can be moved over to additional poller and get the changes done. By doing this you will eventually reach there. If your device admin says it can't be easily done that is when you get in your architect or senior folks who can make the call.

You did raise a request with SW vendor to provide a health check report - as a follow up question always ask them for best practices and they will help you with it.

Hope this helps.

__PRESENT__PRESENT__PRESENT__PRESENT

ImNishat

Thank you for the comprehensive explanation.

The situation is that nodes are being onboarded by various individuals who were advised to select APE over MPE during onboarding. However, they have overlooked this and onboarded the nodes into MPE. Whenever I onboard nodes, I consistently choose APE, though I do it infrequently. I have instructed the other team leader to switch the polling engine of the few onboarded nodes from MPE to APE and am awaiting their action.

To address this issue, I attempted to set APE as the default, but I couldn't find an option to do so. Therefore, I have submitted it as a feature request.

Since most of the onboarded nodes are accessible from both MPE and APE, I believe moving some nodes to APE should not cause any problems, in reference to your fourth point.

If my understanding is incorrect as described above, please let me know if I can conduct myself the necessary testing as outlined in point 4.

chad.every

FPS (Flows Per Second) stem from netflow data collection. It is not something that changes by reassigning nodes from one engine to another engine. This is a configuration on potentially several network devices. In your SolarWinds web console, i'd navigate to My Dashboards > Netflow > NTA Summary page. There is a widget called 'Flow and CBQoS Sources' that shows what devices are sending flow that are ALSO managed nodes in the tool. There is also another widget called 'Last 25 Traffic Analyzer Events' that would report on IP addresses that are sending flow that are NOT managed nodes.

I would guess that you have duplicate flows being received with a FPS count that high. Netflow works very differently than SNMP Polling. Netflow should be setup only on major ingress/egress points on your network like a core firewall or a core router.

In any environment that does send a lot of netflow data, i recommend that you use a Network Load Balancer and configure a VIP to send all flow data too. Behind that VIP, add each SolarWinds engine as a pool member and round robin the flow collection. Down the road, this makes it much easier to scale out as yo simply add more engines behind the VIP and you don't have to go back and reconfigure network devices individually to point to a different netflow exporter IP.

vinay.by

@chad.every - Thank you, that's a great explanation. What approach do we take if a Network Load Balancer can't be deployed ?

stibi

Hi,

The numbers in the table seems incorrect. Really large companies have something like 800K FPS and we are talking about ISP companies. I will take a look at how the FPS(24) are calculated, but for now I would simply ignore that.
Is this causing any issue at the moment? Like how much CPU is NetFlow Service consuming? With such numbers it should be at 100% CPU utilization even on environment with 32 CPUs. Do you see any gaps in the charts or missing data as that would suggest that NTA is not able to store the data and so on.

ImNishat

Hi @chad.every - I have checked your suggested points, please see the below snapshot FYR.

Please guide me, How can I monitor FPS drop rates in NTA?

I need to check whether NLB is available or not in this organization. Because I am not sure about it, to be honest, I don't know about NLB before I ask about NLB, I need to learn more about it so that I can answer any questions about it confidently.

ImNishat

Hi,@stibi - Please see the below report for all three MPE, APE & DB server CPU & Memory utilization for the last 7 days.

stibi

Hi @ImNishat ,

That looks fine and your actual FPS are 15K flows per second on average. I am investigating why we are reporting such large number in the report.

To validate the number of FPS you can look on the performance counters. Open Start -> type and open perfmon -> on the left navigation panel click on 'Performance Monitor' -> click on green 'add' button -> add 'SolarWinds NetFlow' counters. There you will find "PDUs received per second".

Since you have ticket already opened we will continue to assist you through it.

Regards,

Petr

ImNishat

Hi, @stibi - I have performed the suggested steps, Please see the below snapshot FYR. Please tell me if anything you need further from my side.

verwin_u

Hi @ImNishat

@chad.every and @vinay.by have a correct suggestion. You might mis interpreted it.

You need to coordinate this concern to your network team to change the "ip-flow destination" to balance the netflow sources (netqork devices) to your flow collector (mpe or ape).

Ref: https://documentation.solarwinds.com/en/success_center/nta/content/nta-setting-up-network-devices-to-export-netflow-data-manually-sw75.htm

You cam also clean up some sources and interfaces that does not sends netflow data.

Ref: https://documentation.solarwinds.com/en/success_center/nta/content/nta-deleting-flow-sources-and-cbqos-enabled-devices-sw237.htm

Another link here is the best practise on handling your solarwinds NTA.

Ref: https://documentation.solarwinds.com/en/success_center/nta/content/getting-started-guide/nta-flow-environment-recommendations.htm

I also highly recomment you to work with sw support and sw academy to fully understand netflow. Netflow is totally different from snmp and it has huge interaction with your device configuration.

Hope this help and clarify things.

stibi

Hi,

Issue was identified on Palo Alto side.

Every NetFlow v9 packet carries information about number of PDU/flow records as described here https://www.rfc-editor.org/rfc/rfc3954.txt page 9, field Count. Problem here is that Palo Alto started to report incorrect PDU/flow record count:

Regards,

Petr

ImNishat

Hi @stibi

While replying to you, I am getting the below error as shown in the screenshot.