Odd issue after password change

Getting an odd issue with Powershell monitors since changing password whereby it will return "PowerShell script error. Starting a command on the remote server failed with the following error message : Access is denied" on a poll but next poll it works and flaps between these failed and successful states.  These monitors have been in use without issue on the same servers for years. 

I have performed quite a bit of testing around this and can not pin-point the cause. Affects all pollers, both WMI and Agent monitored nodes, encrypted and unencrypted WMI, with/without winrm enabled on node, using specific creds on template and inherited from node. Have removed monitors and reapplied. Have bumped up timeouts. The cred has been removed entirely from the keyring of windows credentials in Orion and replaced with a new one with new title, reapplying to all nodes and the application monitor components. Have removed profile, SID etc from the endpoint servers themselves. 

Has anyone else come across a similar issue after password change?

  • Hey!

    Not sure what version of Orion/Sam you're running but we're running SolarWinds Observability 2022.3 and have a case (Case # 01179175) where SolarWinds support has confirmed an issue with Powershell impersonation that causes Powershell monitors to "flap" up and down. They have acknowledged the issue and said that their internal team is working on a fix but do not have an ETA on said fix. Looking at 2022.4 bug fix list it does not look like it is included in that upcoming version sadly.

    The workaround they provided me is to adjust the Execution mode to "Local Host" instead of the default "Remote Host". I can confirm this works on SOME Powershell monitors but have also run into a case with some of our custom monitors that it does not work running locally. Hoping this gets fixed soon as it impacts a large amount of our monitors. Hope this information and the workaround helps out!

  • and - My first thought is Ugh. When you changed the password, how much of your environment was polling. When I need to update an account or password, I try to stop polling with that account first. Put another way, I shut down all polling servers, and then update the accoutn on the primary poller. Then restart things. Yeah - this is done during a maintenance window but it avoids windows domain controllers and the polling engines fighting over the account. 

    My other thought is the ticket that zdean opened, and how painful that really is. I am getting a bad feeling about 2022.3 as it may have some bugs... I hope that gets fixed soon. I have heard 2022.4 RC2 is out there somewhere, and 2022.4 GA version should be released soon. And there is a new hotfix that may be coming to everyone 2020.2.6 and newer. That last update may be security updates, but they often have a few bug fixes in them. Saying a small prayer to thwack that this gets resolved for you, as it sounds like you are not alone with this issue. 

  • I've reopened my case and can confirm the fix was not included in 2022.4, however was provided another work around from support if anyone would like to give it a try:

    I found an article internally it appears to be a workaround on the issue, You can try and see if that help address the concern we are encountering.

    1. Find and open the following with notepad or other text editors (may need to run as administrator): Program Files\SolarWinds\Orion\SolarWinds.APM.Probes.dll.config or Program Filesx86\SolarWinds\Orion\SolarWinds.APM.Probes.dll.config
    2. In the line: <add key="UseLegacyPowerShellImpersonation" value="false" />change value to <add key="UseLegacyPowerShellImpersonation" value="true" />.
  • @zdean, thanks for sharing. I have applied that change to all our pollers and restarted services for good measure. I will let that soak for a day or-so and let you know whether thats worked for me.

  • 2 days later and lookin' good! thanks @zdean it would seem that this has resolved our issue.