Ability for the SolarWinds Suite to automatically restart agents when they go offline

dilbert1234

We experience issues with agent connectivity where we need to restart the agent processes to resolve. Based on a thread from  @dhinagar_jSolarwinds Agent Issue RCA & Troubleshooting - Forum - Server & Application Monitor (SAM) - THWACK - SolarWinds Community I would like to see a feature added to the platform to resolve the agent connectivity issues. 

The process outlined would be to identify the scope of the Alert based on  

1. Agent connect status in the 'Manage agents' view

2. If no data collected for node in last 10-20 mins based on 'LastSystemUpTimePollUt' attribute using SWQL query.

The Action item would be to restart the process / service based on agent type. One idea would be to integrate  Ansible open source tools to perform Agent Re-Start,

For windows - Restarting Solarwinds agent windows service

For Linux - Systemctl restart swiagent

gkjono

Hi @dilbert1234,

I did this with an alert. I started with an advanced SWQL alert condition that looks for online agents that haven't updated CPU load stats in 180 minutes. That time can be adjusted though.



I then added a trigger action to call a PowerShell script which restarts the agent via a SWIS API call. You have to pass the AgentID to the script as an argument and handle authentication, but the main PowerShell command to do this is:

Invoke-SwisVerb -SwisConnection $Swis -EntityName Orion.AgentManagement.Agent -Verb RestartAgent -Arguments @($AgentID)

I also created an escalation trigger action to move the agent to another polling engine if the action above doesn't solve the issue. As a part o the script it calls, I pull polling engine stats to determine the polling engine with the least amount of load, and then move the agent there with with the command below.

Invoke-SwisVerb -SwisConnection $Swis -EntityName Orion.AgentManagement.Agent -Verb AssignToEngine -Arguments @($AgentID, $($TargetPollingEngine.EngineID))

I had a support case open for six months or so where we were troubleshooting why the agents would stop communicating even though they showed as connected. Below is a fix that resolved 95% of my agent issues.

# Edit following file "c:\Program Files (x86)\Common Files\SolarWinds\JobEngine.v2\SWJobEngineSvc2.exe.config"
# Seek for option jobEngineSettings_Full and add there option emptyRouterCleanupInterval="-00:00:00.0010000" so it looks like this

<jobEngineSettings_Full maxWorkerProcess="10"
jobExecutionRetention="01:00:00"
jobExecutionEnabled="false"
thresholdStreamByteThreshold="131072"
maxWorkerProcessLogFileAge="5.00:00:00"
maxHungJobs="1"
jobHungTimeout="00:00:30"
maxJobThreadsPerWorkerProcess="100"
maxJobThreadsPerWorkerProcess64="1000"
workerMonitoringPeriod="00:00:01.0000000"
forceSeparateWorkersOnAgent="true"
jobErrorSuppressionInterval="01:00:00"
jobErrorSuppressionCleanupInterval="00:05:00"
emptyRouterCleanupInterval="-00:00:00.0010000"
/>

3. Restart service JobEngine v2

This has to be done on all engines (you can do it on one and then copy/paste that file as it's similar on all engines).