Need help with hanging reports

Hi everyone,

I'm looking for a collective solution to our issue as all Solarwinds Support can do is ask for diagnostic reports...., again and again!

Here's the breakdown

We have a ton of custom web-based reports with multiple dynamic query builder queries. It's also worth noting that we have a good number of web-based reports with multiple SWQL queries in them.
We recently upgraded from Solarwinds build 2023.1 RC1 to general release Solarwinds build 2023.1.
Now all reports with multiple dynamic query builder queries or multiple SWQL queries just hang when we open them.
I'm able to reproduce this issue by building a super simple report.
- The report opens successfully when I limit the number of queries to 4 or less.
- The report hangs and won't open when I hit the 5 or more mark.
  - It's worth nothing these can be 5 different data-source queries or 5 of the same data-source queries.
  - It's also worth nothing that the report hangs no matter if all 5 queries are the same or different.
- I'm able to successfully open the report once I delete the queries back down to 4 or less.
Here's what I see in the OrionWeb.log when the report hangs:
- 2023-03-31 13:34:27,049 [312] (71) WARN ASP.global_asax - (null) Long request time: [url:https://10.132.163.34/Orion/Report.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4][time:7683]
  2023-03-31 13:36:23,811 [308] (77) WARN ASP.global_asax - (null) Long request time: [url:https://10.132.163.34/Orion/Reports/Preview.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4&waitid=0744d160ee3f4196ae99bd8558376df2][time:5921]
  *** Assembly App_Web_nodemanagement.asmx.43409ddd, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  *** Assembly App_Web_nodemanagement.asmx.193e71da, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  *** Assembly SolarWinds.Orion.Discovery.Contract, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  *** Assembly SolarWinds.Orion.Core.SharedCredentials, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  *** Assembly App_Web_nodeproperties.ascx.891fea64, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  *** Assembly SolarWinds.UDT.Common, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  *** Assembly SolarWinds.SysMan.Utils, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
  2023-03-31 14:00:53,843 [306] (79) WARN ASP.global_asax - (null) Long request time: [url:https://10.132.163.34/Orion/Reports/Preview.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4&waitid=0744d160ee3f4196ae99bd8558376df2][time:19456]
I found several Solarwinds KB articles that talked about ASP.global_asax warning messages, but non of them really cover the same issue as we're experiencing.
Again I asked SW support for their option based on my findings and all I got was "please send another diagnostic report" .

At this point I feel I can get better support from you all on Thwack, Reddit or a random homeless person on the street than from Solarwinds support!!!!

Now I know the routine where someone from SW jumps on says "case # please", so before you ask it's #01311386.

I also know there's a tendency to censor THWACK posts so, I'm going to post this on the Reddit /Solarwinds.

Find more posts tagged with

2023.1

help

custom support

Reporting

Accepted answers

All comments

jholmes

I'd be curious if you put your 5 queries in a modern dashboard (each as a separate resource) do you get the same issue? If so it may indicate an issue with the backend of SWIS/SWQL queries in some odd limitations. If not its likely just something in the report engine, unfortunately long request times are hard to track down if its actual performance or not. If the modern dashboard loads fine you could try to up timeouts on your reports to see if that affects it.

bourlis

Jeremy, great question! I just checked and there are no issues with the modern dashboards that I've built. I have MD with 12 different queries that opens nice and quick with no issues.

Yes this one is a bugger!!

Here's what I have for Report timeouts.

For SWNetPerfMon.db it's the default 90:

! Database Command timeout in seconds

CommandTimeout=90

and web.config is default exec time off 600 for everything except the schedule which is 300

jholmes

You could try to up them again, the article they have for this has you set those values to 3600, outside of that I'm not on 2023 to be able to see if I can repro it. I'd keep pushing in support to see if you can get to one of the good techs.

support.solarwinds.com/.../Large-report-error-Unexpected-Website-Error-Request-timed-out

jholmes

Coffee is still kicking in just had a workaround idea. If these queries load fine for a modern dashboard can you just set your report to run using the URL for that dashboard?

wluther

@bourlis Would you be willing to export the report and dump it here. Maybe we can pull it into a completely different system and test it there too. I just rebuilt my dev environment this morning, running 2023.1, and could test it here to see if I get the same thing.

wluther

I created a new report and added a custom query to it, then duplicated it until I had 5 of them. The report seems to load just fine for me, but the query is about as simple and calm as could be, so maybe it's just not using enough juice to encounter the issue.

SELECT TOP 1 n.Caption FROM Orion.Nodes AS n

bourlis

Nope, I wouldn't mind exporting/uploading the report, but I'm note sure where too.

But, here's the simple breakdown of the data source for the report. It's a fairly simple SWQL query, I'm sure you can swap out the custom properties for anything.

SELECT
[data].[Vendor],
[data].[CustomProperties]._Disposition
FROM orion.nodes AS data
WHERE
((([data].[Applications].[Components].[Name]) = ('Days passed from last Window Update')) AND (([data].[Applications].[Components].[CurrentStatistics].[ComponentStatisticData]) > (30)) AND (([data].[UnManaged]) != (True)))
WITH NOLOCK

For the resources it's disposition and count based on vendor. It's also sorted by count in a descending order. So that it looks like this:

The report runs when I have 4 of these and hangs when I have 5 or more. However, if I run it as a scheduled report and select run now it will successfully email the page in a PDF format with 5 or more queries.

bourlis

Also just to cover all of my bases and any possible recessed RC 2023.1 oddities, the layout of the report is 3 columns with the default width of 960 px.

wluther

bourlis

Liam, when I hit export report it saves it local to my system as an xml file.

monitoringlife

[quote userid="122564" url="~/product-forums/the-orion-platform/f/report-lab/98491/need-help-with-hanging-reports"]I also know there's a tendency to censor THWACK posts so, I'm going to post this on the Reddit /Solarwinds.[/quote]

Not to sidebar/interrupt, but I'v not heard this before and would like to know more (not challenging just interested in something I've not heard before).

Seashore

I'm actually having the same issue. Running 2023.1. But here it's a couple of reports with two SWQL queries. When opening the reports in GUI it just hangs. If I remove one query the report opens fast. I also today found that if I schedule the full report it also works fine.

The SWQL's runs in less than 10 sec in SWQL studio.

Your_Daddy

Set the folder (C:\ProgramData\SolarWinds\Orion\reporting.tempfiles) permissions (Full Control) for the SolarWinds Orion Application Pool user.

schrumma

I noticed all of my reports did not have any data. I have a ton of SLA reports for website monitors. I had to go back into each report and reselect what I was reporting on.