Getting more ThroughPut out of your Solarwinds Pollers

Have you ever wondered why there is plenty of system resources (CPU/Memory/Network/Disk) on your Solarwinds Poller box but still the throughput is not where you need it to be?

From my years of being a C-Programmer (Sockets) and knowledge of the ISO 7 Layer stack, it was apparent to me that the issue was in the Transport layer.   This post will explain how-to tune the Transport layer to maximize the TCP throughput.

First, to prove this was the case, I wrote a quick Powershell script to list the TCP Connection Counts for each of the Solarwinds Pollers in our monitoring environment.   We were having "unpredictable results" when the Pollers were too busy.   This occurred when the Total TCP connections per poller were in the 12,000 to 15,000 range.    The symptoms of this issue were, like I stated, "Unpredictable Solarwinds results" (i.e. random failures) while there were plenty of resources (CPU/Memory) on the poller box.    That is where my powershell script came in handy to periodically check the TCP Connection counts.    As suspected, the bottleneck was in Layer 4 (the Transport Layer) where the number of TCP connections were not closing quick enough.    By making a simple change (described below) to the TCP settings, it allows the TCP connections to close faster and thus you get a higher TCP throughput and the issue of the "unpredictable results" went away.    Granted, there may be a future time when the TCP connections start to fail again; at which time we may look at purchasing another poller license.   

After the TCP Kernel parameters are changed and you have restarted Solarwinds then run your powershell script for checking the TCP Connection counts again.   You should see a drastic decrease in the number of connections as they are now closing quicker.

Bottom line, if you want to squeeze the most throughput out of a poller, look into tuning your TCP parameters.     This worked for us.   Disclaimer; as with any kernel change you will need to perform your own due deligence and testing for your own environment.

On Windows platforms, if the following tcp parameters are not explicitly defined in the regedit tables then the default values will be used.   If the default timeout is 120 seconds and the maximum number of ports is approximately 4,000, resulting in a maximum rate of 33 connections per second. If your index has four partitions, each search requires four ports, which provides a maximum query rate of 8.3 queries per second.

(maximum ports/timeout period)/number of partitions = maximum query rate

If this rate is exceeded, you may see failures as the supply of TCP/IP ports is exhausted. Symptoms include drops in throughput and errors indicating failed network connections. You can diagnose this problem by observing the system while it is under load, using the netstat utility provided on most operating systems.

Changing the TCP parameters involves two (2) steps:

Step #1: Configure the TCP settings for the server

To set TcpTimedWaitDelay (TIME_WAIT):

  1. Use the regedit command to access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey.
  2. Create a new REG_DWORD value named TcpTimedWaitDelay.
  3. Set the value to 30.
  4. Stop and restart the system.

NOTE:  TcpTimedWaitDelay will not work unless the StrictTimeWaitSeqCheck is set to 1.

To set MaxUserPort (ephemeral port range):

  1. Use the regedit command to access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey.
  2. Create a new REG_DWORD value named MaxUserPort.
  3. Set this value to 32768.
  4. Stop and restart the system.

 

 

Step #2 – Reboot the server

Reboot of server is required after these changes.

keenb

Good Ol Sweet Sweet SWAG

Posted by keenb Expert Mar 8, 2019

Oh, happy days!  After much posting, answering questions and stuff, the day has come, I have finally gotten the coveted THWACK backpack!  This thing is SUPER nice!  It is going to be my everyday carry for sure!  Thanks to everyone like KMSigma and other SolarWinds staffers for providing this great place for us to share ideas and rewarding us with some sweet, sweet, SWAG!

Sascha Giese

Gisec in Dubai

Posted by Sascha Giese Employee Mar 5, 2019

Hi there!

 

SolarWinds is attending GISEC in Dubai between April 1 – 3 this year.
GISEC is the number-one security-focused tech conference in the Middle East, and we’ll be there with our regional distributor Spire.

 

What can you expect from SolarWinds?
Our resident expert Tony Johnson will be there to show you the latest features we’ve added to the Orion® Platform, and how they can help to increase both security and compliance next to help ensure your IT infrastructure is behaving well.
Rumour has it that there might be loads of new features by early April.

Also, you’ll have a chance to meet Venkatesh Ayyala, who works for Spire and lives in Dubai. Venkatesh and his colleagues are providing professional services to SolarWinds customers in the region, from simple configurations to Orion SDK-based customizations.

 

The event itself takes place in the Dubai WTC, which is the same location as GITEX.
If you’ve never been to Dubai, it’s well worth a visit. Last October I took this lovely shot from level 148, only 555 meters above the ground:

 

Enjoy!

 

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.