How do you manage your SolarWinds Agent updates?

robert.poreba

Hi,

I'm raising this topic, as I'm looking for the best way to manage SolarWinds Agent updates, considering the changes in version 2023.3.
Hopefully some of you had similar challenges and maybe came up with some good way of doing it.

As some of you may be aware, there were some critical bugs in SolarWinds Agent in the past. The worst ones we came across were the Agent causing high CPU usage on servers, which was killing the applications running on them and causing outage. It was quite difficult to explain to customers that the software that was supposed to improve uptime, was actually reducing it..

To mitigate this risk, our process so far was to update the Orion platform first, and then gradually update Agents, starting with dev servers first, then doing production few weeks later etc.

This worked well for few years, but in the last version (2023.3), we found that after the upgrade of the platform, previous versions of Windows-based Agents would no longer report any performance data (cpu, memory, disk). We found that it all works after updating the Agent to the latest version, but considering the above risks, we decided not to do that, and we have rolled-back the update.

I have since worked with SolarWinds Support, Success Manager and Product team representative, and they all confirmed that upgrading to 2023.3 requires immediate update of Agents to make them work. Developers also said that this is the only supported way in general (even for previous versions, although they worked ok). There is also no guarantee that this is the only time we'll have to update Agents immediately after platform updates, it could be that in future this process will be similar too..

We really think the Agents should be developed to support at least the N-1 version of the Platform, to give people time to update Agents at slower pace, and mitigate any bugs, but doesn't look like this is the case in the SW design.

I can also imagine there are companies out there with thousands of Agent-based servers, many of them business-critical, and I find it hard to believe that they would be updating all those agents immediately, without proper testing them first.

So my question is, considering the above, how do you manage your Agent updates to mitigate any bugs/risks?
Have you got any suggestions to the problem I described?

Thanks a lot, I appreciate any answers

Kind regard,
Rob

brscott

I have not upgraded to 2023.3 yet, I'm still on 2022.4 and will be for at least 6 monts.  So up to this point the agent has worked 2 even 3 versions back, with the catch that it's a fully installed and working version of the agent.  IF you try to install an older version of the agent, then there might be issues because the modules are downloaded from the polling engine (or is it the web server?) at the most current version.

As for my process.  All of the "non-production" systems are set to autoupdate.  Even before the platform upgrade is complete, once, the main polling engine is upgraded, I am pretty sure that 20% agents are already trying to upgrade also.

Everything else is a manual process, where I push them out as I have time and confidence in the agent, sometimes over months.  Usually there are a few that I never get to.  Even now I see 8 that are still on back dated versions all the way back to 2020.2.6.

Just to be clear, I would not let 200+ agents autopatch (even if they are non-production) if I had not already done it in a test environment.