SolarWinds products have always been agentless. Being agentless was one of the primary drivers for adopting it nearly 7 years ago and has been a powerful argument for many clients in the years since, but sometimes you just need a client. Such was the case when we started to deploy Windows servers into a cloud service. Sure, we could have poked a million holes in the firewall to allow WMI-based data collection and SAM polling but the agent was so much easier.
Enter PowerShell scripts.
While testing out our use cases we ran into trouble when trying to execute a simple PowerShell script-based application monitor. A simple script to check for NTP time drift on a Windows server is below, but let's focus on the following line:
$remoteServerTime = Get-WmiObject Win32_UTCTime -ComputerName ${IP} -Credential ${CREDENTIAL};
When running a test against an agent-based node you get the following error:
Output: ==============================================
Message: Unable to Get-WmiObject Win32_UTCTime
Errors: ==============================================
Get-WmiObject : User credentials cannot be used for local connections
At line:15 char:29
+ $remoteServerTime = Get-WmiObject Win32_UTCTime -ComputerName 127.0.0.1 ...
Hmm, that's interesting - User credentials cannot be used for local connections. After a little testing we realized that removing -Credential ${CREDENTIAL} allowed the script to work as desired. The catch is that not passing that ${CREDENTIAL} variable breaks the script for any non-agent-based node.
Why?
The variable ${CREDENTIAL} grabs the credential set that was assigned to the application monitor at the time the template was assigned to the node. An agent-based node doesn't have a set of credentials assigned to it, instead it runs scripts and other data collection locally on the node via the agent. How do we know this? Look at the returned message
Get-WmiObject Win32_UTCTime -ComputerName 127.0.0.1
Instead of the PowerShell script being run on the polling engine to which the node was assigned the script was passed to the agent and it was executed locally. If it was executed on the polling engine we would have seen something like
Get-WmiObject Win32_UTCTime -ComputerName 10.0.0.10
We tried playing with the Execution Mode, changing it from Local Host (meaning the polling engine) to Remote Host but the error message was still the same.
If you've figured out a way to get a PowerShell script to run on both agent and agentless nodes within a single script I'd love to take a look at what you've done.
As promised, here is the NTP drift script in its entirety. The argument passed is the IP or DNS name of the NTP server to be queried be used in the comparison.
***
$ntpServer = $args[0];
if ( !$ntpServer )
{
Write-Host "Message: Can't find ""ntpServer"" argument. Check documentation.";
exit 1;
}
$ntpQuery = Invoke-Expression "w32tm /monitor /computers:$ntpServer" | Out-String;
$findSkew = [regex]"(?:NTP\: )(?<Value>/?[^s]+)";
If (($ntpQuery -match $findSkew))
{
$ntpToSolarSkew = $Matches['Value'];
$Error.Clear();
try
{
$remoteServerTime = Get-WmiObject Win32_UTCTime -ComputerName ${IP} -Credential ${CREDENTIAL};
}
catch
{
Write-Host "Message: Unable to Get-WmiObject Win32_UTCTime on remote client. $($Error[0])";
exit 1;
}
$localTime = $(Get-Date).ToUniversalTime();
$remoteToSolarSkew = New-TimeSpan $localTime $(Get-Date -year $remoteServerTime.Year -month $remoteServerTime.Month -day $remoteServerTime.Day -hour $remoteServerTime.Hour -minute $remoteServerTime.Minute -second $remoteServerTime.Second);
If ($remoteToSolarSkew)
{
$Skew = $remoteToSolarSkew.TotalSeconds - $ntpToSolarSkew;
$symb=$Skew.ToString().Substring(0,1);
if ($symb -eq "-")
{
$stat=[math]::round($Skew,2);
$tmp=$stat.ToString().Remove(0,1);
Write-Host "Message: Clock drift: $stat.";
Write-Host "Statistic: $tmp";
exit 0;
}
else
{
$stat=[math]::round($Skew,2);
Write-Host "Message: Clock drift: $symb $stat s.";
Write-Host "Statistic: $stat";
exit 0;
}
}
Else
{
Write-Host "Message: Unable to Get-WmiObject Win32_UTCTime";
exit 1;
}
}
Else
{
Write-Host "Message: Unable to query NTP server $ntpServer.";
exit 1;
}
We are getting ready to rollout thousands of those badboys and now I am curious if I need an extra layer of monitoring...
Thank you very much for the post.