This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Powershell Monitors Unknown, "Failed to deserialize probe settings"

We've been using HolyGuacamole's/antonis.athanasiou's Dell Server Storage Hardware Health script for about two years now without any problems. Last week, we updated everything to do with our NMS, migrating to: MS Server 2016, SQL 2016, Orion Platform 2018.2 HF5, NPM 12.3, SAM 6.6.1 (10/31/18 Edit: Updated SAM to 6.7), and setting Orion to HTTPS instead of HTTP.

Almost everything transferred over unharmed except for these Dell HDD monitors. Whereas they used to just work, now they're all constantly flapping between being in an "unknown" state and being "up". If I test the script from the template, it'll work fine 100% of the time.

The error it throws in the application's detail page is

Unexpected error occurred. Failed to deserialize probe settings, probe skipped. Original Exception message is: Cannot access a closed file..

and in C:\ProgramData\SolarWinds\Logs\APM\ApplicationLogs it shows

[STP Pool:66 Thread #1] [C23522] ERROR SolarWinds.APM.Probes.ProbeBase`1 - Failed to deserialize probe settings.

System.ObjectDisposedException: Cannot access a closed file.

I haven't found many hits on this, but I've tried what I've found, which includes:

- Unassigning the application template from all devices, then reassigning it

- Adding [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true} to the scripts

- Changing HttpClientHelperSettings RetryCount - I didn't go through with this because none of the options mentioned in that post were in my config file, so I figured it was depreciated.

- Running the configuration wizard to remove the Check for Publishers Certificate Revocation option - This option wasn't in my configuration wizard

Any ideas what's going on here?

Thanks

SW Dell Storage Hardware Health 1.2 error.PNG

  • I am having the same issues with some of my PowerShell script monitors as well.  Anyone have any suggestions?  We are getting false alerts for these everyday

    Running on Orion 2018.2 and SAM 6.6.1

  • Sorry to see you're having the problem too. Did it just show up, or have you recently run an update too?

    Unfortunately, I made this post over a month ago, so it doesn't look like anyone has a fix, but I have submitted a ticket to see if SW support has any ideas.

  • Have you opened up a support ticket for this issue?

  • Yes, Case 00208193. Gathering up some requested information for it right now.

  • Support thinks it's a problem with the template and not a SW software issue, so I guess we'll live in this limbo until someone smarter than me runs into this problem and figures out how to fix it.

  • I haven't tested this on 2016 server and unfortunately I dont have any means on testing it at the moment.

    Are you using local or remote execution mode?

  • Local execution.

    The case continues: Support gave me a test script (below) to create a test template, and it's doing the same thing (bouncing between "up" and "unknown", saying "failed to deserialize probe"). So maybe it is a platform issue.

    $proc = (Get-Process | Sort-Object -Property CPU -Descending)[0];
    $info = [String]::Format("PID: {0}; Name: {1}; Company: {2}", $proc.Id, $proc.ProcessName, $proc.Company);

    Write-Host "Message.Cpu: Cpu - $($info)";
    Write-Host "Statistic.Cpu: $($proc.CPU)";

    Write-Host "Message.PM: PM - $($info)";
    Write-Host "Statistic.PM: $($proc.PM)";

    Write-Host "Message.VM: VM - $($info)";
    Write-Host "Statistic.VM: $($proc.VM)";

    Exit(0);



  • This seems either a platform or environmental, in either case I would suggest to ask to escalate the case to either an Application Engineer or to development, support will be able to guide you better!

  • We figured it out!

    Per support's request, we tried updating to SAM 6.7.0 HF6, but this didn't help. However, during the process we found a folder on the C: of our Orion server "C:\debug". Within this folder was a sub-folder for every day of the year since we migrated to this new Server 2016 server, and each sub-folder contained thousand of text files titled "PowerShell_transcript.HCCMONITOR.__[random string]" (see first image). It appears the server was logging information and the output of each PowerShell script was run on it. The source of this logging was a GPO called "Server 2016 Defaults" (we did not enable this GPO ourselves). After we disabled this GPO (or possibly just one policy from it-- see second image) on our Domain Controller, the PowerShell logs stopped being written, and now all of our PowerShell monitors are running successfully.

    SW_Logs.PNG

    SW_GPO.PNG

  • Nice job finding that jvogel! I'm not sure that will work for us though, as we use the PowerShell transcription. It's too bad something like logging script output causes an alert to go off.