Solarwinds Website throwing error after Windows Server Migration to 2022

I migrated my Windows Server where Solarwinds Services were hosted and ever since the instance has been far from stable . We often get below errors and it really hard to get anything done

Case is already opened and one of the engineer suggested to do an upgrade to 2023.4.1 but that did not help . We have been getting below error , the case is escalated to DEV team but have not heard back yet , it looks like below . This is just not it , often times we are still on the page and cannot clink on any link and then suddenly web browser returns error for website . Has anyone experienced this or can help on this ?

Find more posts tagged with

network_monitoring

website_error

Accepted answers

All comments

bobmarley

I have seen that, is it just happening on a single resource or several of them? Do you own SAM as it may help to troubleshoot the issue. In the Solarwinds Event Log are you getting any errors about Not able to open a new database connection?

heldy

It happens on any webpage of this instance , especially if its opened for more than 5 minutes .Our license is HCO now we so have everything included , currently we are not monitoring anything with SAM.When filtering logs with being set to as critical , can see below error

lynchnigel

Is the CPU high, is the database residing on the same server? have you looked at ports to check that they are all open as per the list if the database is elsewhere ?

Solarwinds Ports List

heldy

THe CPU is mostly at 45%

Polling engines are not at capacity

Ports are open

SQL database is on a different server

I have two tickets open with TAC on this but no resolution so far , can someone help here , one of the cases have been open for months , i was told DEV team is looking into it but no communication , i have been just collecting logs and submitting those , thats it

sum_giais

Has support directed you in anyway to rebuild the website and/or run the configuration wizard? I'd run through these steps in this order... but that error you show about "Data loss imminent" seems to point to something very wrong in the database. What's it show for more details there? Does it link to a KB at all?

It's tough to know exactly where to start here, especially with limited info but hopefully the first 3 suggestions below, could possibly guide you towards what the issue may be and potentially show you what step of the rest below, may be best suited (or all of them).

Good luck and hope this helps.

1) Review for errors the "C:\ProgramData\SolarWinds\Logs\Orion\swdebugmaintenance.log" log file - this is the log for database maintenance which runs by default at 2:15am

2) Review the "C:\ProgramData\SolarWinds\Logs\Orion\OrionWeb.log" log file

3) Run Active Diagnostics by going to Start -> SolarWinds Platform -> Active Diagnostics Launcher, run all checks and review the results

4) Attempt to run database maintenance manually: support.solarwinds.com/.../How-to-run-the-database-maintenance-manually-in-Orion-environment

5) SolarWinds Platform Permissions Checker: support.solarwinds.com/.../Check-permissions-using-the-Orion-Permission-Checker

6) Run Configuration Wizard, and review the log of it afterward in the latest "C:\ProgramData\SolarWinds\Logs\Orion\ConfigurationWizard.log" log file. Any errors / issues?

7) Rebuild the SolarWinds Platform website: support.solarwinds.com/.../Rebuild-the-Orion-Website

If you get stuck, and need more verbose logs you could adjust specific logging levels if needed: https://support.solarwinds.com/SuccessCenter/s/article/Adjust-logging-levels-with-the-Orion-Log-Adjuster?language=en_US, but be cautious with this and be sure to set them back to default after.

heldy

Did all this but no significant errors , the engineer i was working with on this did some changes after consulting with their dev team , optimizing webpages by decreasing widgets and decreasing nvchar for Node Custom properties but no change , slowness still persists

sum_giais

I'd try turning some log levels up to verbose, to start on OrionWeb and replicate the issue.

From there, potentially enable hubble temporarily to see if it helps identify any issues (it's a longshot tbh) https://support.solarwinds.com/SuccessCenter/s/article/Turn-on-and-use-Hubble and https://support.solarwinds.com/SuccessCenter/s/article/Other-Time-is-High?language=en_US

Could ask support about clearing the information service subscriptions - Website inconsistencies, alerting issues, or other problematic Orion modules and features (this KB is inaccessible currently, not sure why...) but the basic process is as follows:

Disable HA if applicable
Shutdown ALL solarwinds platform services (including dependencies, HA service etc., EVERYTHING on all platform servers)

Run the following SQL query:

UPDATE Limitations SET WhereClause = REPLACE(REPLACE(REPLACE(CAST(WhereClause as varchar(max)), '( (', ' ( ( '), '((', ' ( ( '),'))',' ) ) ')
DELETE FROM Limitations WHERE WhereClause = '1=1'
DELETE FROM LimitationSnapShots
DELETE FROM ContainerMemberSnapshots
DELETE FROM PendingNotifications
DELETE FROM SubscriptionTags
DELETE FROM Subscriptions WHERE EndpointAddress NOT LIKE 'http%'

Start services back up

Back to the basics - have you rebooted the servers? Restarted services?

Have you considered maybe uninstalling and reinstalling the SolarWinds Platform completely on the system? support.solarwinds.com/.../Uninstall-all-Orion-products-from-a-server-and-prepare-for-reinstall