App Pool issue (status unknown) - Related to Component Count being Polled?

bjsusol

We recently ran into an issue where some of our app pools are not showing a status.  The State says started, but status is unknown.

I have a ticket open and we were told that the issue may be related to our component count when polling. We are currently at 11925 and were told we would have problems over 10000.

We are trying to understand what these components are? What does this count consist of? Is there a way for us to reduce the overhead of what’s being polled?

Any suggestions or insight would be appreciated. 

mesverrum

That might be garbage advice from support, or it could be valid. I say that because SAM polling is super complicated, and there isn't a well defined capacity limit, there are some ballpark guidelines that estimate 10,000 monitors on a poller, but just think about the fact that there are 49 different kinds of components and some of them are very simple, others are significantly more complex.  Some scale up really well and others don't.  For example, any time you use an appinsight template you are going to be adding hundreds of components per server, sometimes even into the thousands.  The Appinsight's basically have scripts running under the covers and they are more complicated than a single port test, but they also scale out in a way where it is less of a load than initiating 500 separate SQL user experience monitors.

This is one of the reports I use to eyeball my polling capacity

SELECT e.ServerName, e.IP, e.ServerType, concat(scale.PropertyValue,'x') as [Stacked Pollers], npm.PropertyValue as [NPM Polling %],e.Elements, e.Nodes, e.Interfaces, e.Volumes, sam.PropertyValue as [SAM Polling %], comp.SAMComponents, appins.AppInsightComponents, e.PollingCompletion, e.MinutesSinceKeepAlive,   
CASE  
WHEN PollingCompletion < 98 THEN '/Orion/images/ActiveAlerts/Serious.png'  
WHEN PollingCompletion > 98 THEN '/Orion/images/ActiveAlerts/Check.png'  
END AS [_IconFor_PollingCompletion]  
, CASE  
WHEN MinutesSinceKeepAlive > 2 THEN '/Orion/images/ActiveAlerts/Serious.png'  
WHEN MinutesSinceKeepAlive <2 THEN '/Orion/images/ActiveAlerts/Check.png'  
END AS [_IconFor_MinutesSinceKeepAlive]  
,CASE  
WHEN npm.PropertyValue> 90 THEN '/Orion/images/StatusIcons/Small-Critical.gif'  
WHEN npm.PropertyValue> 75 THEN '/Orion/images/StatusIcons/Small-Warning.gif'  
ELSE '/Orion/images/StatusIcons/Small-Up.gif'  
END AS [_IconFor_NPM Polling %],CASE  
WHEN sam.PropertyValue> 90 THEN '/Orion/images/StatusIcons/Small-Critical.gif'  
WHEN sam.PropertyValue> 75 THEN '/Orion/images/StatusIcons/Small-Warning.gif'  
ELSE '/Orion/images/StatusIcons/Small-Up.gif'  
END AS [_IconFor_SAM Polling %]  
,'/Orion/Admin/Details/Engines.aspx' as [_linkfor_ServerName]  
FROM Orion.Engines e  
left join Orion.EngineProperties npm on e.EngineID = npm.EngineID and npm.PropertyName='Orion.Standard.Polling'  
left join Orion.EngineProperties sam on e.EngineID = sam.EngineID and sam.PropertyName='APM.Components.Polling'  
left join Orion.EngineProperties scale on e.EngineID = scale.EngineID and scale.PropertyName='Scale Licenses'  
left join (select n.engineid,count(n.Applications.Components.ComponentID) as SAMComponents  
from orion.nodes n  
where n.Applications.Template.Name not like 'appinsight%'  
group by n.EngineID) comp on comp.engineid=e.EngineID  
left join (select n.engineid,count(n.Applications.Components.ComponentID) as AppInsightComponents  
from orion.nodes n  
where n.Applications.Template.Name like 'appinsight%'  
group by n.EngineID) appins on appins.engineid=e.EngineID  

order by ServerName

You can see I split out AppInsight components as being separate from "normal" components because experience has proven they don't have the same weight (also you can see im not using any appinsight right now but thats another topic).

Also you can see one of my pollers has ~13,000 components and is reporting at 83% load, but right next to it is a poller with almost 12,000 components that only checks in at 37% loaded.  That has to do with the mix of component types and the number of polling errors on my servers.  I've found components that take a long time to test, generally, seem to be the same ones that impact the SAM loads the most.  Not surprising if you think of the polling engine in terms of having a certain number of time slots to make requests.  The longer the polling job takes the more timeslots it eats up. Additionally I mention errors because having templates assigned that use the wrong credentials, or trying to connect to ports that are blocked at the firewall and are timing out all trigger retries in the SAM job scheduler, which eats up even more slots before we can move on to the rest of the tasks on our list.  I've seen that by getting my credentials and templates sorted out I can reduce the displayed load here.  

I've worked in environments where we ran a lot of the Appinsight templates and it showed us as having 20-30,000 components on a poller, but still only at 25% SAM load, so the components by themselves are really a poor indicator of what's really going on.  

I generally start troubleshooting this kind of thing by going to the main SAM page and just looking at that pie chart of application statuses.  I start with the ones that are down, make sure I don't have templates that are down due my own mistakes, then get into the templates that are unknown and start picking them apart to verify things like having the right credentials, polling methods, url's whatever to ensure I'm not inducing my own headaches.  Once I feel reasonably comfortable with that list I hit the log files on the poller directly.  For SAM stuff you generally will be looking at ProgramData\SolarWinds\Logs\APM and it's children.The brute force way is just to sort by last modified and start by checking out whatever is happening right now.  If something is broken you can expect it is being written to the log constantly.  Looking at mine right now I see things like:

The remote name could not be resolved: 'some server shortname that probably needs to be an fqdn'

Connection Attempt to x.x.x.x as wrongdomain\serviceaccount Failed. Attempt 1 of 3

Every time I sit down and clear up these kinds of mistakes my polling load ends up reducing and my pollers have fewer problems, and my dashboards end up with less things showing as down/unknown so I look slightly more competent.  That bad credential is using up 3x more time slots on my poller than it would if I gave it the right credential so the poller didnt have to keep retrying all day.  If you poke around in the logs long enough you'll probably run into exactly the nodes you are having trouble with and might get a more specific clue as to what is wrong with them.

If you are a powershell user this is the script I use to sift through my logs so I don't have to spend so much time scrolling through a jungle of messages
https://github.com/Mesverrum/MyPublicWork/blob/master/OrionLogParser.ps1