HA Service Monitoring in AWS via an ELB using PowerShell

Oh, the title didn't scare you off yet, eh? Perfect. Welcome to my latest challenge!

We developed an HA monitoring strategy for our on-premise applications where we check for the status of a service using a VIP. This allows us to monitor an HA-protected service without caring whether or not the back-end cluster is an active/active or active/passive cluster or whether it is a cluster of 50 servers. As long as one of the services responds (or the web page responds, etc. etc.) then we know that the business service is available. Yes, I know, it doesn't tell us if the service is *degraded* but that is another ball of wax.

Enter AWS.

We were asked to monitor an application cluster in AWS. Load-balancing is done via ELBs in AWS. We are also testing the use of the SolarWinds agent based on some rather direct advice from aLTeReGo and the team of product managers at SolarWinds. I know that I have the security groups set up correctly because I am able to query the status of the service directly to the servers and I have the same SG applied to the ELB. I have configured a listener on the ELB for 5985 and 5986, but when I try and test across the ELB I get an RPC Not Listening error.

Here is the simple PowerShell script I am using. Remember, these servers are monitored in NPM via the agent, not via WMI or SNMP.

$service = get-wmiobject -query "SELECT State FROM Win32_Service WHERE Name='SolarWindsAgent64'";

$status=$service.state

Write-Host $status

switch ($status)

{

{$status -eq 'Running'} {Write-Host 'Message: Service is '$status; Write-Host 'Statistic: 0'; break}

# If the service status is 'Running' then returns a statistic equal to the exit code (0) exits as Up.

default {Write-Host 'Message: Service is ' $status; Write-Host 'Statistic: 1';}

# If the process is not running, return a statistic of 1 (matching the exit code) and exit as Down.

}

Execution mode is remote host. If I wanted to run as local host I would need to modify the query to include the target and credential.

$service = get-wmiobject -query "SELECT State FROM Win32_Service WHERE Name='SolarWindsAgent64'" -ComputerName '${IP}' -Credential '${CREDENTIAL}';

I know that WinRM is working direct to the servers, but whether I run this in local or remote mode across the LB, using just the -ComputerName or both -ComputerName and -Credential, it doesn't seem to work.

I get that this is really an AWS question and not an SAM monitor question, but if you've managed to figure this one out I'd be glad to hear what you did to make it work.

And, yes, I did try and do a wide-open AWS security group for both the ELB and target nodes. It doesn't help.

Find more posts tagged with

high_availability

AWS

PowerShell

elb

Accepted answers

jbiggley

So I did some more testing. This appears to be either an AWS ELB fail or an ELB configuration fail.

I flipped the AWS instances to SNMP and verified I could connect from the polling engine to the instances without issue. Remember, I am using the same security group for the ELB as I am for the instances. I also added an HTTP port forward for 161...and it failed.

I *suspect* that ELBs need to see some sort of HTTP or HTTPS header in order to forward the packet. Given that SNMP is UDP 161 then we won't get getting the magic of ELB.

TLDR; AWS ELB /= F5 BIG-IP but there is a F5 BIG-IQ on AWS -- F5 BIG-IQ Virtual Edition for AWS (BYOL) on AWS Marketplace

All comments

adatole

I have to admit, the title is REALLY daunting.

I'll also admit, I don't have an answer. But I can't wait to see the discussion the rest of our loyal THWACK-izens bring to the table!

KMSigma.SWI

Couple things that I would check first:

Can you authenticate any other way directly to either of the cluster nodes from the Orion Server?

Try Enter-PSSession -ComputerName (AWSComputerName) -Credential ( Get-Credential -Message "AWS Local Admin Creds" )

Try Invoke-Command -ComputerName (AWSComputerName) -Credential ( Get-Credential -Message "AWS Local Admin Creds" )

Other thing to verify is that the IP of the Orion Server is listed as a trusted endpoint in the WSMan configuration settings.

I'd also change the script so that it runs with slightly less memory:

$Status = Get-Service -DisplayName "SolarWinds Agent" | Select-Object -ExpandProperty Status

if ( $Status )

{

Write-Host "Message: Service is $Status"

Write-Host "Statistic: $( [int]( $Status -ne "Running" ) )"

}

else

{

Write-Host "Message: Unable to Query Service"

Write-Host "Statistic: 5"

}

mrxinu

Ditto to what KMSigma said above - when you're connecting to those hosts directly vs through the ELB does your source IP address vary? More specifically, does the latter put you on a different source subnet than the destination?

jbiggley

First, the level of awesomeness on this reply is beyond awesomesauce level. Now to the nitty-gritty details.

Yes, I can Enter-PSSession -ComputerName fqdn -Credential (Get-Credential domain/user) and connect direct to the AWS instance from the polling engine.

Yes, the polling engine is permitted in the TrustedHosts. (Found via PowerShell using winrm get winrm/config) << Placed this here for posterity sake

And, yes, I know it is bad form to have TrustHosts = * -- this is a lab environment

2016-09-08 16_28_30-10.22.2.112 - Remote Desktop Connection.png

When I run Get-Service -ComputerName fqdn -DisplayName "SolarWinds Agent" | Select-Object -ExpandProperty Status; Write-Host $Status against the server itself from the polling engine I get a response, but if I try and run it against the ELB I get and error:

Get-Service : Cannot open Service Control Manager on computer 'ELB-NAME-HERE'. This operation might require other privileges.

I verified that I can get to both members of this HA pool from the polling engine. Did I mention these nodes are being monitored using the agent? Not that is should matter when I am running the command from PowerShell on the polling engine, but it's worth mentioning. And, yes, the same security group applied to the target servers is applied to the ELB. Just to rule out that I flubbed the rule creation on the ELB

jbiggley

I find that more and more of my posts here start with some level of ominous overtone. It's like I've collected on the easy things, now its on to the things that make your brain hurt!

jbiggley

Source IP, no. In our case it doesn't matter because the security groups allow all of our core subnets to AWS. We're lucky enough to have AWS as a LAN extension (VPN connected and via DirectConnect coming soon) so I don't have to worry about that pesky thing called the Internet

Yes, we are definitely on a different subnet source to destination, but since I can get from the polling engine to the target servers (see my response to KMSigma) then I should be able to get through the ELB (on the same subnet as the servers -- remember, this is a lab...best practices be damned!) but it feels like I am stopping there.

I wonder if it has to do with the way that AWS ELBs work. I wonder if they are NATing connections to the targets?? Although that shouldn't matter anyway as we have TrustedHosts = * (again, lab -- security be damned!)

KMSigma.SWI

jbiggley - you totally did everything the same way that I would have. I'm thinking that it must be a quirk of the AWS ELB. Maybe it's trying to round-robin or something?

Regardless, I don't think that this is a SolarWinds Orion or SolarWinds Agent problem. I might just set up the same to do some testing. If I do, I'll let you know what I come up with.

jbiggley

you totally did everything the same way that I would have.

Well that's the best compliment ever!

I'm going to flip those nodes to SNMP or WMI and see if it makes a difference just to make sure it isn't some oddity, but I agree -- this certainly seems like an ELB oddity.

jbiggley

So I did some more testing. This appears to be either an AWS ELB fail or an ELB configuration fail.

I *suspect* that ELBs need to see some sort of HTTP or HTTPS header in order to forward the packet. Given that SNMP is UDP 161 then we won't get getting the magic of ELB.

TLDR; AWS ELB /= F5 BIG-IP but there is a F5 BIG-IQ on AWS -- F5 BIG-IQ Virtual Edition for AWS (BYOL) on AWS Marketplace