marcrobinson · Observability Detective · ✭✭✭✭✭

Good job, I think you found your root cause. Those one-off account issues are painful. 

I ran into something similar. The issue was with his AD account actually. We did the same with multiple browsers, incognito mode, clearing browser cache and using different pc's. You can also try a different account with access on this user's system too. If everything works except the affected account - the resolution was…

Custom undp polling should be able to get those values. Create one, apply it to all apc ups. I seem to recall there being sets of APC snmp OIDs also. Each with varying levels of details. But I am thankful that you found an answer, even if it is kinda 'meh.' 

you are correct AppOptics is a separate license. You mentioned prometheus? Will you be monitoring containers then? Check into the SAM container monitoring - Docker Container Monitoring in Solarwinds Orion. IT is Prometheus based per my understanding. 

Check SNMP settings, you could be viewing old data. In other words, snmp polling is failiing. I tend to use custom pollers instead of the built in ups polling, but that is what I have noticed when there is a difference between the two. 

I have had to use IIS logs to track this type of activity. I REALLY would love more audit logs for swis. This is something of a security hole AND a troubleshooting black hole. Netflow is another method that I have tracked some of this. But you are looking at larger amounts of trafffic. 

Not as much. He indicated that it was a known issue resolved in 2019.4. I just skimmed through the orion and npm release notes, but nothing stood out. BUT he asked for snmp walk, and there was a tidbit about not being able to see some metrics for NPM in that release note. I still agree with your thoughts on the overflow…

My first thought is to open a support ticket. It does sound like this may issue may have returned, and having tickets hit support will escalate the issue/final fix. Otherwise, you may have something that they do have a workaround or fix for. 

Some links related to alerting: Fail WMI Polling alert? - Forum - Network Performance Monitor (NPM) - THWACK (solarwinds.com)

It looks like a bug as the issue was escalated to the Dev. They have been working on it. No news yet. If I had to guess - 2023.4 might see the fix. 

THis might be what you are looking for, using a swql query. It is filtering on running state, but ones that are not paused. SELECT c.ContainerImageId, c.ContainerImage.Name as ContainerImage, c.Name as ContainerName, c.IpAddresses, c.State, c.TimeCreated, c.Command, c.Cpu, c.Memory FROM Orion.Cman.Container C WHere state =…

After upgrading to 2023.3 - still seems to be happening. I agree with your analysis above. I just opened a ticket to see if I am missing something else, bad polling jobs, or the devices have a sense of humor when returning metrics. I hope it is not the last as that would mean they are already plotting to take over the…

Thank you for sharing. This is great information for anyone running into this issue. 

Take a look at the 2023.3 which went to public release about two weeks ago. Several of us have run the upgrade and it has been overall pleasant. I just ran it yesterday from 2023.2, and one small hiccup, it went fine. One primary and 8 secondary pollers - so far so good. This version seems to be stable, and the amount of…

Try 2023.3 or 2023.2.2. Both include udates to the job engines. 2023.3 seems to be the most stable recent release. 

FYI - the list resources is more of a discovery tool for adding resources to monitoring. You can choose which detected items to actively monitor. 

SWIS REST API Port Deprecation, did you know? - Orion SDK - The Orion Platform - THWACK (solarwinds.com) I believe the latest version of swql studio defaults to 17774 now. But the image shows no response on 17777 - and the message sounds a lot like a firewall issue. I believe that ports 17777 and 17774 are both supported…

From the manage nodes screen, click the [+] or triangle symbol in front of the node. That will list all the resources. You can check individual resources and then click on Delete in the menu bar. Just make sure that do NOT have the node also checked. It will delete the node and everything with it. I If you wanted to…

I may need to correct myself. The agent might not be able to run a undp (universal device poller) job. The kb I checked was a bit old, but probably still relevant. Otherwise - SAM template as noted below. 

If you are able to run wireshark and look at the traffic logs, that would eliminate some confusion. Otherwise, I am starting to think that the orion logs may be useful. Have you run SolarWinds Active Diagnostics on one of the new pollers? Just choose I want to select the tests and leave it at default (all). It will take a…

@"bobmarley" - What he said! I am doing similar using custom sql queries to get status of each cluster. Who is primary/secondary, memberstate, replication sync state, etc. can be all scooped up. Those are added to a SAM template and applied as needed. 

Secondary monitoring then is the next level of fun. That can vary based on your needs, requirements, and capabilities/tools at your disposal. 

These are momentary disconnects. The system will often be able to eek out an alert even if it gets worse. Too much worse, then you are correct, solarwinds will be effectively stopped. If you need immediate notification - then alert on one event per polling period. I suspect that it could be chatty, but if every solarwinds…

try allowing 17778 and 17774. Then my thoughts turn to looking at those new servers directly and potentially running wireshark. 

I think @monitoringlife is indicating that creating a custom polling job via the undp poller and assigning it to that node will cause the agent to run that job. Otherwise, I agree that you cannot really modify the agent itself. 

I have to ask, are you testing from the servers or your workstation as you may get differing results from the tnc command. (Network firewalls etc)

Consider monitoring the virtual ip/dns pointer via Appinsight too. You get a current and consistent data feed. The cluster members just need "normal" monitoring. This does mean that you will need to monitor that vip as a node. AppInsight will not generally detect failovers, but you can get that information other ways.…

Its a windows event log component that you would create and add to a template. I have a template with just this and one other event monitored. This event is located in the App and Services logs, not the windows logs. Seeing a LOT of these is an indicator that communication to the sql server is not happening. You may see…

That is a good question with several responses. If polling is off, but database checkin from the pollers is good, you are looking at potential issues with nodes. I will give a short overview here and try to paste in links from thwack later. Short overview is that you can have reports and even alerts if snmp or wmi is…

No corruption that I have experienced, same with lockups. If SQL is having issues, this monitor will most likely indicate it or even experience it - letting you know that SQL wants some love and attention. To be fair, appinsight is not a full blown sql monitor, but it does a great job overall. Especially as it comes with…