Managing remote PowerShell v 3.0 hosts

I've just spent a few days banging my head of the keyboard trying to get a simple powershell script working across our environment and having sussed the problems thought I'd share and try to ease someone else's pain.

 

The problem manifested itself when my fully working monitor template worked on some server and failed horribly on others. The monitors get returned the lovely "No Output" message, telling you the square root of bugger all. Even when I copied the scripts over and run them locally they were still failing.

 

I found the Solarwinds Technical Reference Guide: Configuring and Integrating PowerShell which has the commnads for allowing remote connections, but the later part of the command (instructing the host to allow connection from the Orion server) failed with syntax errors:

 

> winrm set winrm/confog/client @{TrustedHosts="Orion IP address"}

     Error: Invalid use of command line. Type "winrm -?" for help.

 

Searching of these errors proved pretty damn fruitless, as did the reasons to why it might be failing, until I checked the versions:

 

> $PSVersionTable


Name                           Value

----                           -----

WSManStackVersion              3.0

PSCompatibleVersions           {1.0, 2.0, 3.0}

SerializationVersion           1.1.0.1

BuildVersion                   6.2.9200.16481

PSVersion                           3.0 <-- Not V 2.0 as required by APM PowerShell.

CLRVersion                     4.0.30319.1008

PSRemotingProtocolVersion      2.2

 

By luck I came across a TechNet article on "How to start a remote session with the Windows PowerShell 2.0 Engine"

     Starting the Windows PowerShell 2.0 Engine

This detailed how you could force incoming connection to use a specific version of PowerShell and within seconds, my remote scripts were working!

 

2> register-PSSessionConfiguration -Name PS2 -PSVersion 2.0

 

     WARNING: Register-PSSessionConfiguration restarts the WinRM service and all dependent services.

     All WinRM sessions connected to Windows PowerShell session configurations, such as Microsoft.PowerShell and session configurations that are created with the Register-PSSessionConfiguration      cmdlet, are disconnected.

 

     Confirm

     Are you sure you want to perform this action?

     Performing operation "Register-PSSessionConfiguration" on Target "Name: PS2 <blah>

     This will allow selected users to remotely run Windows PowerShell commands on this computer".

     [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

 

Once done my scripts ran error free!

 

Hope it helps someone else!

  • Did you have to do anything else?  I successfully registered that on the target machine (Server 2016).  However, when I try to test the component within the "Edit Script" window I get "Not Defined", and if I test it via "Test Node" I get "Test failed with 'Down' status on <MachineName>".

    Did you have to modify anything on the SolarWinds server as well?

    I've also tried a number of configurations of the "Use HTTPS Protocol" (checked and unchecked) and the "Port Number" (5985 and 5986 for HTTP and HTTPS, respectively).  No dice.

  • Worth noting that I can otherwise Invoke-Command from each server to the other, al la "Invoke-Command -ComputerName <MachineName> -ScriptBlock {dir C:\}".

  • Are you passing credentials correctly from the template? When running it locally it will use your current users, when running it from the template it won't.

    Failing that, post up the script and I will have a look for you

  • I've tried not passing creds, passing the node's creds, passing other creds that I know are a local admin on the target; no dice.

    The script is a test one I threw together specifically just to test remote execution (it's querying scheduled task status; Orion already has better functionality built in, but again; this is just to test something).

    -----

    $tasks = Get-ScheduledTask -TaskPath "\" | Where {($_.State -ne "Disabled") -and ($_.TaskName -notlike "User_Feed_Sync*")} | Get-ScheduledTaskInfo | Select TaskName,TaskPath,LastTaskResult,NumberofMissedRuns

    $errorcount = 0

    $errorlist = @()

    ForEach ($task in $tasks) {

        If ($task.LastTaskResult -ne "0") {

            $errorcount++

            $errorlist = $errorlist + $task.TaskName + "<br>"

        }

    }

    Write-Host "Statistic.ScheduledTaskErrors: $errorcount"

    Write-Host "Message.ScheduledTaskErrors: $errorlist"

  • Okay, copied it over, tested it and it fails on the message formatting.

    Commenting out the last Write-Host line, however, gets it working again, as did just removing the + '<br>'.

    So  I swapped your <BR> for a colon, and the script is running from a Windows Powershell Component.

    pastedImage_0.png

    So here's my version, that is working for me, I shortened the unique description as well, just in case:

    $tasks = Get-ScheduledTask -TaskPath "\" | Where {($_.State -ne "Disabled") -and ($_.TaskName -notlike "User_Feed_Sync*")} | Get-ScheduledTaskInfo | Select TaskName,TaskPath,LastTaskResult,NumberofMissedRuns

    $errorcount = 0

    $errorlist = @()

    ForEach ($task in $tasks) {

        If ($task.LastTaskResult -ne "0") {

            $errorcount++

            $errorlist = $errorlist + $task.TaskName + ":"

        }

    }

    Write-Host "Statistic.SchedErrors: $errorcount"

    Write-Host "Message.SchedErrors: $errorlist"

  • I copied in your edited script, but am still unable to get things working.  I have since tried running a PowerShell window on the SolarWinds server as the same credentials used in the template, and with it I can execute remote PowerShell commands to the same target servers that the component monitor in SolarWinds is struggling to reach.  Is there a guide or documentation that I can follow to comprehensively go through the setup required to get Remote Execution working?  I feel like something just isn't set up correctly, but I'm starting to grasp at straws as to what that might be.

  • Okay, so I am guessing that there some other security settings inhibiting things then....

    Have run through this page: Problems using the Windows PowerShell Monitor in SAM - SolarWinds Worldwide, LLC. Help and Support

    Especially the bit around running "winrm quickconfig" on the host or adding it (the Orion server) as a trusted host, on the polled server? 

    winrm quickconfig

    winrm/config/client@{TrustedHosts="Orion-Server"}

    Also have a look at this page:

    Troubleshooting WMI issue in SAM - SolarWinds Worldwide, LLC. Help and Support

    And

    Test WMI services using WBEMTest issue in SAM - SolarWinds Worldwide, LLC. Help and Support

    There is also a debug option, in the template under the Advanced options. When enabled it will log to ProgramData\SolarWinds\Logs\APM\ApplicationLogs\AppID<ApplicationID>

    This also might help.

  • Ah, I didn't know where those logs went, so that's a big help.  I checked that out, and sure enough to our theory I see the following:

    [When trying via "Use HTTPS Protocol" unchecked and port 5985]

    "2019-02-12 10:15:31,322 [STP SmartThreadPool Thread #15] [C5554] ERROR SolarWinds.APM.Probes.PowerShellProbe - System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server X.X.X.X failed with the following error message : The WinRM client cannot process the request. Default authentication may be used with an IP address under the following conditions: the transport is HTTPS or the destination is in the TrustedHosts list, and explicit credentials are provided. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. For more information on how to set TrustedHosts run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic."

    [When trying via "Use HTTPS Protocol" checked and port 5986]

    "2019-02-12 10:15:43,604 [STP SmartThreadPool Thread #13] [C5554] ERROR SolarWinds.APM.Probes.PowerShellProbe - System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server X.X.X.X failed with the following error message : The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig". For more information, see the about_Remote_Troubleshooting Help topic."

    Logging into that X.X.X.X server, running "winrm quickconfig" results in "WinRM service is already running on this machine.  WinRM is already set up for remote management on this computer."

    After running "winrm set winrm/config/client '@{TrustedHosts="OrionServer"}'" I get the following output:

    [Target Server WinRM Config]

    Client

        NetworkDelayms = 5000

        URLPrefix = wsman

        AllowUnencrypted = false

        Auth

            Basic = true

            Digest = true

            Kerberos = true

            Negotiate = true

            Certificate = true

            CredSSP = false

        DefaultPorts

            HTTP = 5985

            HTTPS = 5986

        TrustedHosts = OrionServer

    Still, a subsequent test fails.  I also tried adding the target server as a TrustedHost on the Orion server; which still fails.

    [Orion Server WinRM Config]

    Client

        NetworkDelayms = 5000

        URLPrefix = wsman

        AllowUnencrypted = false

        Auth

            Basic = true

            Digest = true

            Kerberos = true

            Negotiate = true

            Certificate = true

            CredSSP = false

        DefaultPorts

            HTTP = 5985

            HTTPS = 5986

        TrustedHosts = TargetHostname

  • Okay, proceed to the webmtest, if this works from your Orion server to the polled server, then the PowerShell should also.

    And there isn't any other firewalls or port restrictions that could be causing this?  You'll need RPC/DCE with dynamic returns allowed.

  • Actually, I think I just fixed it.  I shy of saying that I figured it out, but I did get it working in this test case.

    I dug around thinking along the lines of the HTTP error calling out "Default authentication may be used with an IP address under the following conditions: the transport is HTTPS or the destination is in the TrustedHosts list, and explicit credentials are provided."  In that digging I found that when using HTTP with WinRM that (by default at least) the connection is only allowed by using host names.  IP addresses can't be used, and it's an IP address that SolarWinds is passing ("Connecting to remote server X.X.X.X failed...")

    I then added the IP address of the SolarWinds server as a TrustedHost to the target server, no dice.  Figuring I may need to set up both ends of that connection I then set the Orion server's TrustedHosts to "*" (thinking that I'll for now at least need this to be able to reach other servers.)

    The subsequent test of HTTP remote execution was successful.  I still need to dig around a little more to make sure I properly understand this, and the implications of these settings, but thank you so much for getting me on the right track.