Tools to check for windows server port congestion??

We have an on-prem Orion instance.

1 main engine, 8 additional polling engines, a beefy SQL server, and 4 additional web servers. We seem to be having issue where Solarwinds is reporting latency on external nodes and elements that do not really have latency when checked from other sources. I am beginning to think we are experiencing port congestion. But this is very difficult to troubleshoot because we need to monitor when the issues occur. But, we don;t know the issues occurred until AFTER they have passed.

Does anyone have troubleshooting advice or know of any app/tool that could help isolate an issue such as this occurring in our NPM environment?

We are running 2023.2

Find more posts tagged with

port congestion

reporting latency

Accepted answers

All comments

monitoringlife

I think there is an out of the box SAM template for this "Windows TCP Port Exhaustion". If its not out of the box just ran this command:

netstat -ano | find /i "estab" /c

Edit: I apply this to all my SW servers as we ran into this issue ourselves. It also also recommend to expand the ephemeral port range, KB: Ephemeral Port Exhaustion On Solarwinds Orion Servers

marcrobinson

Agreed. Also check latency/port exhaustion to your SQL server. There are some windows events to look for, in particular Event ID 4001 from SWService. The message will be something like the system is unable to open a connection to the sql server. Since those are logged in windows events, you can search after the fact for them. AND create a SAM monitor for them too. When a bunch start occurring - time to look at what is happening to the sql server and your path to it.

leonejames1

Interesting I have no collector.businesslayer logs

monitoringlife

KB might be out of date, I don't see them named that on 2023.2.

@KMSigma.SWIcan you point us in the right direction of the log (and poke someone internal to update the KB).

Ref:

[quote userid="60518" url="~/product-forums/the-orion-platform/f/forum/98796/tools-to-check-for-windows-server-port-congestion/310828#310828"]KB: Ephemeral Port Exhaustion On Solarwinds Orion Servers[/quote]

KMSigma.SWI

Best way to get the KB's updated is to leave feedback on them directly. (There should be a little happy/sad man you can click on and leave notes.)

Did the 2023.2 release not ship with an updated Application Template for SolarWinds Platform? I just recently upgraded my own, so I haven't looked yet.

adam.beedell

Stick an alert on your polling engines being over a threshold, and have them all monitoring themselves and each other.

I'd agree that it's probably port exhaustion but could be other stuff too, sticking the test on loopback covers bases while diagnosis continues