Our organization uses various custom properties to help define alerts based on criticality and environments. In the past few years we have identified all production servers by Tier 0 through 4, where 0 is critical infrastructure 1-2 are business critical, and 3-4 are meh. We also further sort based on environment, we have Production, Test, Development, and Training. Then we break down some types, like Citrix, Time Clock's, Downtime PC's, etc..
Recently one of our Tier 1 application owners did not receive an alert when a drive started nearing a critical threshold. Based on the face we want all Production servers, except for the Citrix environment triggering the alert we scoped the alert as follows:
However when we did the Show list we only saw a few dozen objects, when there should be a few hundred. The query looked correct. It should return all servers in Production but were not defined as Citrix, we even tried changing the query to does not contain "Citrix," however it still did not work. So I opened a case with SolarWinds support to figure out what was wrong.
So they called and we setup a remote session, and as we were troubleshooting the aha moment happened. We do not place a type value for all servers, and the query does not treat empty values the same, so the query was returning all servers that had a type defined, but not equal to Citrix.
The solution? Add an and/or block and change the query to also return results for empty as below:
This works perfectly and we now have a functioning alert the way we want..
So the moral of the story, if you are basing queries on properties that could have an empty value, make sure you take the empty into account and query accordingly.