marcrobinson · Observability Detective · ✭✭✭✭✭

@"bshopp" bless you sir! Thank you for responding and letting us know that there is a light at the end of this flow/tunnel. 

I will see if I can replicate this problem tonight.

Thank you for uploading the link to the template documentation. Since you have postman working with same credentials and api calls, this is isolated to your servers/solarwinds installation/configuration. 

I will have to test it again as I am on 2023.1 right now. You should be fine with 2 charts. Three is probably ok, but that was about the limit. I never found an answer otherwise.

Yeah I just saw the notes for 2023.1.1 and 2023.2 - looks like a lot of these issues were picked up and resolved. Thanks for the reply. 

Storage account sounds interesting. The gateway is another ugly thing entirely. I have noticed that the app gateways want the token refreshed hourly, and the api monitor does not seem to have this capability currently. Let me poke around a bit and try to find some other posts to cross reference about Microsoft. The…

@"vinay.by" and @adam.beedall - I agree on the simple additional action. In many cases I get a cc'd (separate email address for this) on most alerts. I also have a daily alert that resets itself which sends email and page at a given time. The trigger is something like - is the main poller on line? I use scheduling to keep…

@"bobmarley" You hit it on the head. That little trick in the node management interface has save me COUNTLESS times when dealing with loopback interfaces, and other common nuisances, umm, interfaces... 

Check to make sure DB maintenance is running too. That will slow things down. Network latency can be another cause. Netpath can be used to help find determine latency root cause. 

WPM and Storage Monitor are not licensed by HCO currently. They still can run and be upgraded in the same orion/solarwinds instance. BUT they still require their current licensing model.

I opened a ticket as I had two separate 2023.1 GA instances showing this issue and wanted to get more confirmation. It is a known bug - fix is planned for later this year, version 2023.3. They also confirmed that no issues are occurring with maintenance otherwise - in particular the retention interval settings. The data is…

Ditto. No RC here, only GA versions. PRoblem did not exist before upgrade to 2023.1 GA

I am seeing this also with 2023.1 in two environments. Both environments were upgraded to 2023.1, the first from 2020.2.6 and the other from 2022.4.1. Sounds like a bug to me. 

Yeah that sounds like a feasible plan. I did a similar migration years ago. We spun up a new instance on trial licensing, got stuff migrated (nodes, credentials, views, alerts, reports) to where we liked, then shutdown the other system as it was redundant and moved the licensing. Please reach out to your account rep if…

oh yeah - [tag:DarkTHEME] Not really aware of any theme capability, I think you could modify the text at one time, but that was about it. BUT THIS IS AN AWESOME IDEA! 

After reading your note, I ran through two instances with multiple APEs and the sticking points were in the 30% range, then again at 55%. I am not sure what is happening, but it 'hung' during each of those phases for about 15-20 minutes each on the main poller. After it passed each of those percentages, it would jump.…

Hmm What version of NTA are running? Are you reporting on both inbound and outbound flows? 

Aside from the logs, there is an Azure App gateway API. you will need to configure an MS GRaph application, grant it permissions, then create the api poller. Currently it looks like the API implementation for the azure app gateways want the token renewed hourly, which is something that I have not found the api poller…

There is the buddy tool - or was. But it requires the sql database to be available, so there are some limitations there too. 

Agreed. You need something to at least ping the primary server until it is online. I would recommend checking it after patching anyways. Windows patches 'never' cause problems. 

I just ran that upgrade, also switched from legacy licensing to HCO. Licensing was the kicker, otherwise the upgrade went fairly smoothly. I have multiple polling engines and web servers. Otherwise, take a look at some of the other thwack postings. Based on general responses in thwack - the 2023.1 has been better overall…

I think that the DB Maintenance error reporting bug from 2022.4 remains. There is one known bug where container entity maintenance misreported. You may have stumbled across one of those errors. Please post if this is the case or if it is a new one. 

Actually, I would argue that 2020.2.X was stable and the 2023.1 GA appears to be stable. I just ran the upgrade from 2020.2 to 2023.1 and other than some weird licensing and windows permission errors during install, it went smooth. I am still validating overnight jobs with NCM. Looks like my mail settings needed to be…

Ran into similar issue with unable to access folder to delete files. I checked and the folders were already empty. Turns out I had to reboot and rerun the installer. Since the uninstall already completed, I was concerned, but the installer picked up right where it needed to be and complete the install. I did call support…

Interesting. Have you looked at using the API to modify/set a custom property, that then triggers an alert? Its roundabout, but would allow you to have the alert actions reset that property too. Its also pretty flexible as all you have to do is set the properties you want and now you can effectively trigger them…

There are some other benefits here such as simply separating the user/scripted rest api traffic. There have been a few threads about controlling access to the api, and this can definitely be used to observe where the interactions are coming from and limit what machines (yes the discussion has gone that far) can access that…

1663027 – net-snmpd double free or corruption error (redhat.com) Here is one of them. 

You would have to download all checks and tag information, then create a script to run api calls against a given group of tags/check numbers from your downloaded list... Ugly to say the least. If you do create it, and are willing to share that, I am sure that the gods of thwack would smile upon you. 

I do similar to track what is going on. If I see a lot of muting - I usually expect some systems to go down for maintenance. Audit logs can also effectively track node adds and node removals. But locking down a user to restrict the API and powershell is limited at best currently. I wish more auditing was done on the api…

The tags show up in the Check Detail for uptime checks. https://api.pingdom.com/api/3.1/checks/<checkNumber> BUt that would be one request per check to get that information. So using tags as a filter? meh not likely from what I have seen.