Wouldn't it be nice if SolarWinds Observability - Self hosted, could alert when it's not working correct?!
Under "My Deployment" you can see all servers health. Let's create an alert on that also! (This will not alert on all issues, but many)
We will create an alert rule that uses a Custom SWQL query on "Polling Engine". Looks like this:
Full query:
-- Mandatory part
SELECT Engines.Uri, Engines.DisplayName FROM Orion.Engines AS Engines
-- User added part
INNER JOIN
(
SELECT
P.HostName
FROM Orion.HA.PoolMembers AS P
WHERE
P.Status>1
) AS P1 ON Engines.DisplayName=P1.Hostname
Then on the action side, it's up to you to add whatever action you want. But below is a custom SWQL variable that will give your alert text some info on what is wrong:
${N=SWQL;M=SELECT
CONCAT('Error# ',Status,': ',StatusMessage) AS T
FROM Orion.HA.PoolMembers
WHERE
Hostname='${N=SwisEntity;M=ServerName}'}Then an alert can look something like this:
Download and import attached file or create it yourself from above instructions