Best way to monitor Solarwinds Agents themselves?

Hey all!

We're starting to get more and more cases of our Solarwinds Agents hanging on some of our servers. Usually it manifests as a hung process or a maxed out CPU core.

How have people approached monitoring the Agents themselves? Due to the somewhat generic name, I'm not having much luck seeing if there are any monitoring templates on Thwack. I could create a simple process monitor, but I'm surprised no one has done that yet or dialed it in.

Is it possible that using SAM to monitor the agents itself is duplicating information somewhere else, or may cause even more performance issues?

Find more posts tagged with

Accepted answers

All comments

bogdan.stan

What i use its a basic report with the below data source. It show agents and SNMP nodes that don't have new data available in the last 5 pooling intervals.

SELECT Nodes.NodeID AS NetObject, Nodes.Caption AS Name, nodes.IP_Address AS IP, Nodes.ObjectSubType as Type, nodes.unmanaged as UnManaged, nodes.machinetype as Model, Nodes.status AS Status
FROM Nodes
Where LastSystemUpTimePollUtc < DATEADD( second, -5*PollInterval, GETUTCDATE()) AND (status =1 OR status =14 OR status =3) AND ObjectSubType IN ('SNMP', 'Agent')

ahbrook

I got this imported into a custom report, but I'm not 100% sure what I'm seeing. I see a list of 23 SNMP nodes and 1 Agent node that are on the list. All of them have "STATUS=1."

@bogdan.stan - what columns do you typically display and/or act on? Like, how do you make it clear to folks that it is showing agents that haven't checked in for a while?

I'm still kind of hoping to find some way to make sure an agent doesn't take up 100% resources for an extended period of time.

the_ben_keen

Yes you can SAM to monitor the service. You build a SAM Template by going to a node that you know has the agent installed then opening Service Control Manager on that server. Once the list populates, locate "SolarWinds Agent and then click "Start monitoring this service". Follow the on-screen instructions to finish building the template and assigning it to the nodes that are using the agent. You can then have an alert trigger and send a message when, for example, the agent services stop running.

ahbrook

This is what I was figuring I'd need to do. It just gives me pause to not see a template out there already. Is this just not something people do?

the_ben_keen

I just uploaded the SAM template that I set up in my home lab.

You can find it by going here.

ahbrook

Thanks! For now I've just thrown in the main service, the processes it generated (Job engine, logging engine, and the agent process itself), and I'm letting data collect. I didn't go for the individual job engine process threads because I figured those would be too noisy.

I'll compare notes.

bogdan.stan

If you want to only display Agent you just need to remove SNMP from the last brackets.

But in your situation i think is better you leave it since you have lots of SNMP pooling problems. You might even considering adding WMI just in case you also have this monitoring method.

ahbrook

Thanks again to everyone who has chimed in so far, both here and in the related thread I opened: https://thwack.solarwinds.com/product-forums/server-application-monitor-sam/f/forum/93231/how-do-i-find-unknown-apps-in-the-connection-monitor-widget

I'm finding more and more evidence that a spike in WmiPrivSE.exe is what is causing the high CPU. It comes and passes in a few seconds, but is likely related to the agent reading.. something. I suspect event logs? I'm not sure. What's annoying is that this (and the Solarwinds job engines) are sub-calls from the agent itself, so it is very hard to capture. I was thinking it might be the antimalware service or another scanner, but the more I think about it and find evidence here on Thwack, the more I think it is just the age of the server (2012R2) and not being cleaned up.