BUG in 2019.4 UPGRADE ??
The agents continue to be a nightmare. We did an upgrade to the platform and went to the latest 2019.4 Hotfix 3.
Being well aware of the Auto-update of agents, we decided to globally turn off this auto-update before we went through with the upgrade. We did have a support case open on standby just in case we had a tough time with the upgrade.
Despite having Disable Agent Auto-update as well as each agent being set to auto-update disable at a DB level (using a SQL query), we have seen many production server agents go into a state: auto-update in progress, some have already completed the update and some are waiting for a restart. Disaster awaits .. coz I am certain that .NET will also be updated to 4.8 and once the server has gone through a power cycle, not sure about application impact due to the .NET upgrade.
Case # - 00346321
Will update on the outcomes of the upgrade.
The most notable agent failure we had after the upgrade was the Agent on Orion itself. I can't for the life of me figure out what I should do to recover. Download the MSI, and run it manually?
The Agent service does not exist on this server any more - not in services.cpl.
The plugin "report" (which is not an Orion report, but I digress ...) shows:
I'm not even certain that the Orion server itself needs to have an Agent, or if this was some experiment involving our other Orion installations that monitor the Orion server.
I'm gonna should all over this, I'm afraid.
I did the 2019.4 upgrade yesterday, and the only agent that shows as being down after the upgrade is on Orion itself! And it has .NET 4.8! How it got 4.8, I don't have a clue. Developers are mute on such matters.
For those that want to check .NET versions:
The Microsoft .NET marketing and engineers were all on serious drugs when they implemented versioning and dependencies, I think. Oh, and the Windows Features feature does not feature (couldn't resist) showing other types of installs of .NET., because those versions are not Windows Features (i.e. Used by Windows itself).
GRIPE: I think the Release Notes are poorly worded regarding the Agent updates. Some things in the Release Notes seem to refer to the agent on Orion servers and polling engines. Other things seem to refer to agents on any server. There's also more than 1 way to install .NET:
I bet there are also different kits for installing and bundling things for OEMs. This is a huge mess for enterprises that must maintain tight versioning and auditing of layered products for things like vulberabilities.
I'm thinking one needs the .NET 4.8 Runtime in most cases. The Setup and Release Notes should direct us. IMO, if you write code in .NET, you need to document .NET for the user. You need to check for .NET for the user, at least on the Orion node. Any
auto update of the agent should fail if the proper .NET is not already there, and roll back to the old agent.
IMO, the Setup should have had a check for .NET 4.8 on Orion itself, and warn us if there is indeed a problem with our Agent Auto-Update setting, and not let the install proceed without confirmation. The rule of thumb should to have setup check dependencies up front, and not count on the installer having read through the Release Notes.
I see no evidence of any update pushed to my other servers, even though Auto-Update was enabled on Orion itself. I have since disabled it. Who knows when Auto goes auto?
I'm now looking into what's wrong with the Agent on Orion itself.
Is this going to be a risky proposition?
2 MAJOR RISKS HERE
RISK1: PRODUCTION SERVERS DEPENDENT ON EXISTING .NET versions The 2019.4 upgrade will not just update the Solarwinds agents but also update the .Net version. This could be disastrous on production servers running an older version of .Net (say 4.0 or 4.5) to the .Net 4.8. The production application may run into trouble due to the .Net upgrade.
RISK2: AGENTS NOT REPORTING PROPERLY Before upgrading, we may want to consider disabling the Auto-upgrade for agents. This means that the agents continue to work on the existing versions. In this case, what is the expected behavior of older agent versions reporting to a newer platform version? Any compatibility issues? What if, the agents do not report properly or do not report at all?
Is this going to be a risky proposition?
Yes. I would strongly suggest updating the .NET Framework first, perform regression testing, and then update the agent. The 4.8.x version of the framework has caused a lot of issues for some users' business applications.
There is a lot of "it depends", but the short answer is yes.
The Agent is upgraded with the Orion Platform version not specifically SAM. So if you just happened to upgrade other modules last month, then no, otherwise yes.
Also each agent has a setting for "Allow automatic agent updates". So it can be done one agent at a time if want to control that setting. Though it's difficult to do in groups. I think I will submit a feature request to enable/disable automatic updates from the manage agents page using the "More actions" drop down.
I have not tried it yet, but there is also a note in the "whats new" for Orion 2019.4 that it will upgrade your .Net version on all agent systems. I'm *assuming*, they mean that it will upgrade the .Net when it upgrades the agent, and not mess with the .net version if I have automatic agent updates turned off, but again, I have not done that yet.
Have you attempted to disable the auto agent update and have a newer Orion platform with older agent versions without the .Net upgrade on agent systems?
Is there some sort of compatibility matrix available?
I have not tried this yet. I don't expect to until late January ( after new years day, I'll work on the exact schedule with my peers). I still haven't figured out how to handle it. I'll probably disable the auto-update on all agents, then after the orion upgrade turn it back on for all 2016+ servers, then do all of the older ones slowly, or not at all. It's going to be a task, and I'll need to see if the API can disable/enable the auto update setting.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.