Menu Failure Error: RunQuery failed, check fault information. Execution Timeout Expired.

Norco_Savage

Hello all,

Some context to this issue: We've been dealing with this error ever since a team member duplicated our Solarwinds server for testing. I think where things went wrong is that this duplicate was still pointed to the SQL server, so there was a time where two servers were writing to the same database.

We're currently trying to work through it with support, but I was wondering if anyone else has run into this issue and what they did to try and fix it. 

Some things we've tried so far with support:

- Upgrading to ver 2024.2.1

- Checking Database maintenance (Finishes with data retention warning, we have retention settings that are longer than default)

- Checking Storage/CPU/Memory usage

- Checking the Websites SQL table (only one entry exists and it is correct)

Running Nartac IIS Crypto is the next step they want us to try.

 Pictures of the error and .txt of the full error output are below.

[View:/cfs-file/__key/communityserver-discussions-components-files/194/Solarwinds_2D00_error.txt:147:321]

esimson3

I had this same issue in a customer environment today where new licensing was implemented and the secondary polling engine was decommissioned.

The likely problem is you have too many pending notifications for the little notification bell in the top menu bar to populate.  The query that is timing out is the query that aggregates and populates that little menu.

The exceptionMessage was accurate in our case, the cause was that the decommissioned polling server's services had gotten restarted and it was generating a huge number of plugin failed to start events.  We found this by going on the Solarwinds SQL server and looking at the latest events in the notification table with this query: 
SELECT TOP (1000) [NotificationID]
      ,[Title]
      ,[Description]
      ,[CreatedAt]
      ,[Ignored]
      ,[NotificationTypeID]
      ,[Url]
      ,[AcknowledgedAt]
      ,[AcknowledgedBy]
  FROM [SolarWindsOrion].[dbo].[NotificationItems]
  Order By CreatedAt Desc

This should give you a big list of notifications.  You can also see this in the Event viewer in Solarwinds, something should be making a ton of frequent events.

We ultimately stopped the services on the old polling server and then deleted all of these specific notifications.  The root cause of the error in the UI is that there are too many events for solarwinds to aggregate by type before the api query times out.  Hope that helps, keep in mind the producer of your events may be different.