Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Agent install with identical MSI and MST files produces different results on two identical servers in the same domain

Hi Folks,

I've been going spare trying to get silent agent installs working properly on several servers which are all in the same domain. Taking apart the Agent install process has been interesting but certain results I'm getting are just bonkers.

This is all in Azure BTW.

From the Agent Download page I created a manually configured MST with the FQDN of our test poller with matching IP address. The test environment is running 2024.1 successfully.

Copied the MSI and MST onto the target servers and install with 'msiexec -i "msiName" TRANSFORMS="mstName" /qn' and a few seconds later I have an Agent running. However, on the first server the install was fine and I got the config file I was expecting and it's talking to the test poller. The second server got a completely different config file that I know had previously been on that server.

Realising the MST contained the certificate with the poller address etc I looked at the certs installed on the first server.

'SolarWinds Agent Provision - <GUID>' must be created when Provisioning succeeds. Solarwinds-Orion is the same on all servers and is the cert that authenticates with the poller.

On the second server I have this:

Solarwinds-Orion is the same as before with identical details. The second one I have no idea, unless that's created by the agent and when provisioning takes place this cert is removed and the 'Provison- <GUID>' one takes its place.

Question is, why should two identical servers act completely differently and where is this incorrect config coming from? It's not the templates or stored in %appdata% that I can see.

Another oddity is if I create an MST with an IP address of 1.2.3.4 and install the agent with it I do get a config file containing that IP address. If I stop the service and change the IP address to one I know is correct and restart, the IP no longer matches the cert and that config file is OVERWRITTEN with the incorrect one which is obviously stored...somewhere. To test this I changed the permissions of the config file so only I could access it and started the service - it complained that the cert didn't match.

I realise I might not be explaining very well, this could explain the radio silence from Support after I logged a ticket last week.

Anyone come across this before?

Cheers

Adrian

Find more posts tagged with

Accepted answers

All comments

radek.slaby

MST file contains the "Agent Provision" and "Solarwinds-Orion" certs (common for all installations) and init file containing info to make the initial provisioning connection.

The provisioning will assign the final "Orion-Agent" cert (unique per agent) and all IPs/hostnames available at the Orion server (to be used by agent in any future connection attempts). This will replace all the info from MST.

Ie. the agent that received the "Orion-Agent" cert is installed and fully provisioned,

while the one remaining on "Agent Provision" is just installed, but not provisioned, ie. not ready for operation. It can still be manually provisioned from Control Panel.

"Solarwinds-Orion" cert is same everywhere - it's a mere copy of the Orion cert that allows the agents to verify authenticity of the server they're connecting to.

fop

Yep, I can see that by dumping the MST. What I can't see is where the SolarWinds.Agent.Service.exe.cfg comes from. On the working node it's a simple file with a single entry for the FQDN and IP address I configured at MST creation. On the non-working node (installed from the same files remember) it's a monster config with 4 entries for the poller, none of which are correct. Therefore that config can't be embedded in the MST and must be hidden somewhere on the non-working node.

fop

Thought I'd add the relevant parts of the two config files in question once I'd sanitised them.

This is from the working node.

and this is from the non-working node:

It's baffling.

fop

Found it. The agent initially connects with the info in the MST and finds the poller correctly. Then for some reason this happens:

24/02/15 15:23:41.258 PID: 10780 TID: 11076 [INFO] AsyncTaskManager::OnClientConnect - Server with device ID [6C8E0057-F2C4-410F-9812-CC80D00B4A23.AMSF] was connected
24/02/15 15:23:41.258 PID: 10780 TID: 4680 [INFO] TaskExecuter::ProcessPolicy - file name [C:\ProgramData\SolarWinds\Agent\tmp\8e40d558-f8ac-4fd7-a2e9-f0b93d0345fd]
24/02/15 15:23:41.258 PID: 10780 TID: 4680 [INFO] ewAsyncRpcCommon::ProcessConfigBase::ApplyPolicy - policy file name [C:\ProgramData\SolarWinds\Agent\tmp\8e40d558-f8ac-4fd7-a2e9-f0b93d0345fd]
24/02/15 15:23:41.263 PID: 10780 TID: 10456 [INFO] ewCore::WaitProvider::Wait - timeout in [ProvisionClientLib::AsyncRpcClientLoopImpl::Loop], elapsed [0.0037607] of [1] ms
24/02/15 15:23:41.263 PID: 10780 TID: 2436 [INFO] ProvisionClientLib::WebSocketClientBase::OnRegisterClient - 0000020D964ED510 - Hub [FilesHub], serverID [6c8e0057-f2c4-410f-9812-cc80d00b4a23]
24/02/15 15:23:41.273 PID: 10780 TID: 4680 [INFO] ewAsyncRpcCommon::ProcessConfigBase::ApplyPolicy - config file name [C:\Program Files (x86)\SolarWinds\Agent\SolarWinds.Agent.Service.exe.cfg], process config flag [1], overwrite flag [Overwrite]
24/02/15 15:23:41.273 PID: 10780 TID: 4680 [INFO] ewAsyncRpcCommon::ProcessConfigBase::ApplyPolicyXml - overwrite flag [Overwrite]
24/02/15 15:23:41.273 PID: 10780 TID: 4680 [INFO] ewAsyncRpcCommon::ProcessConfigBase::ApplyPolicyXml - policy section [target], policy protection [1]
24/02/15 15:23:41.273 PID: 10780 TID: 4680 [INFO] ewAsyncRpcCommon::ProcessConfigBase::ApplyPolicyXml - policy section [target] indicates whole section replacement
24/02/15 15:23:41.273 PID: 10780 TID: 4680 [INFO] ewAsyncRpcCommon::ProcessConfigBase::ApplyPolicyXml - config section [target] was removed

This message is sent from the poller whereupon it overwrites the [target] section of the config with entries from the mothership. Now I just need to work out WHY it happens on one node but not the other.

BTW I always remove the Agent from the Web GUI first while testing. If it's in contact with the agent it removes everything from the target machine including the certificates, but if it can't communicate with the agent and it's removed via 'apps and features' instead then not everything is removed and it has to be done manually.

fop

Hah! Turns out that the node that 'works' doesn't, or didn't. Agent was running and I got the correct response when I went into the Agent GUI, but the logfile showed me it wasn't connecting to the poller, and that turned out to be because the Azure NSG for that particular server didn't allow port 17778 outbound. Slightly embarrassing

So, on successful provision the poller will overwrite the [target] section of the agent's config file with a stub stored here:

wss://poller:17778/solarwinds/agentmanagement/

I now need to find out either how to make it not do that, or change the source config so it has all the connection details it will ever need. We only have 4 pollers after all.

fop

OK, a fix

When an agent is provisioned, the poller it's connected to will send up a system-generated config to overwrite whatever you put in the MST file. I don't believe you can stop this happening.

What you CAN do is add entries to the AgentManagement_PredefinedConnectionSets table which are added to the system generated file. Put your entries in with a higher priority than 0 to make sure they go to the top of the list, and if you set ConnectionSetId to 0 it will become the default for all future agent installs. There's no GUI for this, it has to be done via DB Manager or SQL Studio. Make sure you back up your database before continuing.

We got there!