cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 14

Agent Polling and PowerShell Scripts

SolarWinds products have always been agentless. Being agentless was one of the primary drivers for adopting it nearly 7 years ago and has been a powerful argument for many clients in the years since, but sometimes you just need a client.  Such was the case when we started to deploy Windows servers into a cloud service.  Sure, we could have poked a million holes in the firewall to allow WMI-based data collection and SAM polling but the agent was so much easier.

Enter PowerShell scripts.

While testing out our use cases we ran into trouble when trying to execute a simple PowerShell script-based application monitor.  A simple script to check for NTP time drift on a Windows server is below, but let's focus on the following line:

$remoteServerTime = Get-WmiObject Win32_UTCTime -ComputerName ${IP} -Credential ${CREDENTIAL};

When running a test against an agent-based node you get the following error:

Output: ==============================================

Message: Unable to Get-WmiObject Win32_UTCTime

Errors: ==============================================

Get-WmiObject : User credentials cannot be used for local connections

At line:15 char:29

+ $remoteServerTime = Get-WmiObject Win32_UTCTime -ComputerName 127.0.0.1 ...

Hmm, that's interesting - User credentials cannot be used for local connections.  After a little testing we realized that removing -Credential ${CREDENTIAL} allowed the script to work as desired.  The catch is that not passing that ${CREDENTIAL} variable breaks the script for any non-agent-based node.

Why?

The variable ${CREDENTIAL} grabs the credential set that was assigned to the application monitor at the time the template was assigned to the node.  An agent-based node doesn't have a set of credentials assigned to it, instead it runs scripts and other data collection locally on the node via the agent.  How do we know this?  Look at the returned message

Get-WmiObject Win32_UTCTime -ComputerName 127.0.0.1

Instead of the PowerShell script being run on the polling engine to which the node was assigned the script was passed to the agent and it was executed locally.  If it was executed on the polling engine we would have seen something like

Get-WmiObject Win32_UTCTime -ComputerName 10.0.0.10

We tried playing with the Execution Mode, changing it from Local Host (meaning the polling engine) to Remote Host but the error message was still the same.

If you've figured out a way to get a PowerShell script to run on both agent and agentless nodes within a single script I'd love to take a look at what you've done. 

As promised, here is the NTP drift script in its entirety.  The argument passed is the IP or DNS name of the NTP server to be queried be used in the comparison.

***

$ntpServer = $args[0];

if ( !$ntpServer )

{

Write-Host "Message: Can't find ""ntpServer"" argument. Check documentation.";

exit 1;

}

$ntpQuery  =  Invoke-Expression "w32tm /monitor /computers:$ntpServer" | Out-String;

$findSkew = [regex]"(?:NTP\: )(?<Value>/?[^s]+)";

If (($ntpQuery -match $findSkew))

{

    $ntpToSolarSkew = $Matches['Value'];

    $Error.Clear();

    try

    {

        $remoteServerTime = Get-WmiObject Win32_UTCTime -ComputerName ${IP} -Credential ${CREDENTIAL};

    }

    catch

    {

        Write-Host "Message: Unable to Get-WmiObject Win32_UTCTime on remote client. $($Error[0])";

        exit 1;

    }

    $localTime = $(Get-Date).ToUniversalTime();

    $remoteToSolarSkew = New-TimeSpan $localTime $(Get-Date -year $remoteServerTime.Year -month $remoteServerTime.Month -day $remoteServerTime.Day -hour $remoteServerTime.Hour -minute $remoteServerTime.Minute -second $remoteServerTime.Second);

    If ($remoteToSolarSkew)

    {

        $Skew = $remoteToSolarSkew.TotalSeconds - $ntpToSolarSkew;

  $symb=$Skew.ToString().Substring(0,1);

        if ($symb -eq "-")

        {

            $stat=[math]::round($Skew,2);

            $tmp=$stat.ToString().Remove(0,1);

            Write-Host "Message: Clock drift: $stat.";

            Write-Host "Statistic: $tmp";

            exit 0;

        }

        else

        {

            $stat=[math]::round($Skew,2);

            Write-Host "Message: Clock drift: $symb $stat s.";

            Write-Host "Statistic: $stat";

            exit 0;

        }

    }

    Else

    {

        Write-Host "Message: Unable to Get-WmiObject Win32_UTCTime";

        exit 1;

    }

}

Else

{

    Write-Host "Message: Unable to query NTP server $ntpServer.";

    exit 1;

}

5 Replies
Level 11

i had the same issue and this fixed it for me. I had submitted a ticket to support and they sent me this article. Thank you very much for the post.

Level 10

I know this is an older thread, but I use an if to get around this issue:

[...other stuff...]

$mainAddress = '${IP}'

[...other stuff...]

try {

    if ($mainAddress -notlike '127.0.0.1') {

          $mainOutput = Get-WmiObject -Query $query -ComputerName '${IP}' -Credential $mainCreds }

     else {

          $mainOutput = $mainOutput = Get-WmiObject -Query $query }

}

catch {

    Write-Host "Statistic.FreeSpace: 0"

    Write-Host "Message.FreeSpace: Main Server WMI Query Failed"

    exit

}

0 Kudos
Level 11

The forwarder run as a service, standard running or not monitoring. If you wan to automate a restart after it crashes, you must delete the lock file ([install_location]\var\run\splunk\conf-mutator.pid).

If the service exits gracefully, it removes the file. If the service is running, there is an exclusive lock on the file.

Level 11

OK, here you go...
I ran into the same issue trying to use Get-WMIObject to pull a file object so I could delete a lock file after a Splunk Universial Forwarder service failure on Windows. I am using the procedure to create an encrypted password file on all pollers and using that and ${USER} to build a credential object that i pass to Get-WMIObject:

Using PSCredentials without a prompt

First I made the encrypted PW file in $PROGRAMDATA\SolarWinds with the command:

read-host -assecurestring | convertfrom-securestring | out-file C:\securestring.txt

This prompts for the password and writes a file with the password encrypted in it.

I create a credential object to pass, looking in the environmental variable for c:\programdata, with this:

$Pass = cat $env:programdata\solarwinds\rsa\securestring.txt | convertto-securestring

$Creds = new-object -typename System.Management.Automation.PSCredential -argumentlist $user ,$pass

For the file path for Get-WMIObject to use, we need to change "\" to "\\":

$WMIfileName = $fileName.replace("\","\\")

Here are some code snippets (some of the command line variables are used in parts of the script not included here):

#requires -version 2

<#

.SYNOPSIS

Looks for a service, checks this component state, deletes file and restarts a service if needed.

.DESCRIPTION

Run via SolarWinds' Server & Application (SAM) components.

.INPUTS

CL Arguments:

0 - ${Component.ID} (Static, required, SolarWinds macro based argument)

1 - {${Node.Caption} (Static, required, SolarWinds macro based argument)

2 - ${IP} (Static, required, SolarWinds macro based argument)

3 - ${USER} (Static, required, SolarWinds macro based argument)

4 - Service name (Required)

5 - File to be deleted (optional)

.OUTPUTS

None.

.NOTES

Version: 1.0

Author: Michael Landman <Michael.Landman@FirstCitizens.com>

Creation Date: Jul 14, 2016

Purpose/Change: Initial script development.

#>

#---------------------------------------------------------[Initializations]--------------------------------------------------------

# set error action to silently continue

$ErrorActionPreference = "SilentlyContinue"

#----------------------------------------------------------[Declarations]----------------------------------------------------------

# script version

$scriptVersion = "1.0"

#----------------------------------------------------------[Init variables]--------------------------------------------------------

if (-not ("System.ServiceProcess" -as [type])) {add-type -assemblyName "System.ServiceProcess"}

# Pull CL arguments, the SAM way

$thisComponentID            = $args[0]

[string]$targetServer        = $args[1]

[string]$targetServerIP     = $args[2]

[string]$user                = $args[3]

[string]$serviceName         = $args[4]

[string]$fileName             = $args[5]

$WMIfileName = $fileName.replace("\","\\")

[string]$msg = "Default"

[int]$errLevel = 0

$nowTime = get-date -format t

...

#-----------------------------------------------------------[Functions]-------------------------------------------------------------

# Function to build a credential set from an encrypted file

function BuildCredentials(){

    $Pass = cat $env:programdata\solarwinds\rsa\securestring.txt | convertto-securestring

    $Creds = new-object -typename System.Management.Automation.PSCredential -argumentlist $user ,$pass

    return $Credentials

}

...

$creds = BuildCredentials

{#look for and delete the lock file

$file = Get-WMIObject -Query "Select * From CIM_DataFile Where Name ='$WMIfileName'" -computer $targetServer -credential $creds

  if($file)

    {

     $file.Delete()

    }

# See if the file still exists. If so, exit with error   

$file = Get-WMIObject -Query "Select * From CIM_DataFile Where Name ='$WMIfileName'" -computer $targetServer -credential $creds

if($file)

  {

  $msg = "The {0} file {1} failed to be deleted. No additional action will be taken on {2}. " -f `

  ($serviceName, $fileName, $targetServer)

# report the message and statistic from above

write-host ("Message.ServiceState: {0}" -f $msg | Out-string)

write-host ("Statistic.ServiceState: {0}" -f $errLevel)

exit 0

Tell me more about this issue with Splunk UFs on Windows?    We are getting ready to rollout thousands of those badboys and now I am curious if I need an extra layer of monitoring...

0 Kudos