This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Need help with hanging reports

Hi everyone,

I'm looking for a collective solution to our issue as all Solarwinds Support can do is ask for diagnostic reports...., again and again!

Here's the breakdown

  • We have a ton of custom web-based reports with multiple dynamic query builder queries.  It's also worth noting that we have a good number of web-based reports with multiple SWQL queries in them.
  • We recently upgraded from Solarwinds build 2023.1 RC1 to general release Solarwinds build 2023.1.
  • Now all reports with multiple dynamic query builder queries or multiple SWQL queries just hang when we open them.
  • I'm able to reproduce this issue by building a super simple report.
    • The report opens successfully when I limit the number of queries to 4 or less.
    • The report hangs and won't open when I hit the 5 or more mark.
      • It's worth nothing these can be 5 different data-source queries or 5 of the same data-source queries.
      • It's also worth nothing that the report hangs no matter if all 5 queries are the same or different.
    • I'm able to successfully open the report once I delete the queries back down to 4 or less.
  • Here's what I see in the OrionWeb.log when the report hangs:
    • 2023-03-31 13:34:27,049 [312] (71) WARN  ASP.global_asax - (null)  Long request time: [url:https://10.132.163.34/Orion/Report.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4][time:7683]

      2023-03-31 13:36:23,811 [308] (77) WARN  ASP.global_asax - (null)  Long request time: [url:https://10.132.163.34/Orion/Reports/Preview.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4&waitid=0744d160ee3f4196ae99bd8558376df2][time:5921]

      *** Assembly App_Web_nodemanagement.asmx.43409ddd, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly App_Web_nodemanagement.asmx.193e71da, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.Orion.Discovery.Contract, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.Orion.Core.SharedCredentials, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly App_Web_nodeproperties.ascx.891fea64, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.UDT.Common, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.SysMan.Utils, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      2023-03-31 14:00:53,843 [306] (79) WARN  ASP.global_asax - (null)  Long request time: [url:https://10.132.163.34/Orion/Reports/Preview.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4&waitid=0744d160ee3f4196ae99bd8558376df2][time:19456]

  • I found several Solarwinds KB articles that talked about ASP.global_asax warning messages, but non of them really cover the same issue as we're experiencing.
  • Again I asked SW support for their option based on my findings and all I got was "please send another diagnostic report" . 

At this point I feel I can get better support from you all on Thwack, Reddit or a random homeless person on the street than from Solarwinds support!!!!

Now I know the routine where someone from SW jumps on says "case # please", so before you ask it's #01311386.

I also know there's a tendency to censor THWACK posts so, I'm going to post this on the Reddit /Solarwinds.

  • I'd be curious if you put your 5 queries in a modern dashboard (each as a separate resource) do you get the same issue? If so it may indicate an issue with the backend of SWIS/SWQL queries in some odd limitations. If not its likely just something in the report engine, unfortunately long request times are hard to track down if its actual performance or not. If the modern dashboard loads fine you could try to up timeouts on your reports to see if that affects it. 

  • Jeremy, great question!  I just checked and there are no issues with the modern dashboards that I've built. I have MD with 12 different queries that opens nice and quick with no issues.

    Yes this one is a bugger!!

    Here's what I have for Report timeouts.

     

    For SWNetPerfMon.db it's the default 90:

    ! Database Command timeout in seconds

    CommandTimeout=90

    and web.config is default exec time off 600 for everything except the schedule which is 300

    <location path="Orion/ReportWait.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="600" />
    </system.web>

    <location path="Orion/Report.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="600" />
    </system.web>

    <location path="Orion/Reports/AsExcel.aspx">
    <system.web>
    <httpRuntime maxRequestLength="64384" executionTimeout="600" />
    </system.web>

    <location path="Orion/Reports/Preview.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="600" />
    </system.web>

    <location path="Orion/Reports/Scheduler.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="300" />
    </system.web>

  • You could try to up them again, the article they have for this has you set those values to 3600, outside of that I'm not on 2023 to be able to see if I can repro it. I'd keep pushing in support to see if you can get to one of the good techs. 

    support.solarwinds.com/.../Large-report-error-Unexpected-Website-Error-Request-timed-out

  • Coffee is still kicking in just had a workaround idea. If these queries load fine for a modern dashboard can you just set your report to run using the URL for that dashboard? 

  • I'm sorry I don't quite understand what your suggesting.  Are you suggesting rebuilding the reports as modern dashboards? If so, that would be decent work around option however most users like to have their reports ran on a scheduled bases and mailed.  Also it doesn't appears there's a PDF or Excel export function in MD yet.

  • Would you be willing to export the report and dump it here. Maybe we can pull it into a completely different system and test it there too. I just rebuilt my dev environment this morning, running 2023.1, and could test it here to see if I get the same thing.

  • Ok this is just odd!  First off thank you that's a good work around option and might actually shed some light on the issue. 

    I just ran my test report with 6 queries and to be expected it hung.  I closed it down after waiting 5 or so minutes.  

    I then added the report to a scheduled report and then hit the run now option it was successful.  It emailed me the web page with all 6 queries.

    So with this bit of information, what would you all suggest next?  I'm thinking of upping the report timeout sections in web.config folder.

  • I created a new report and added a custom query to it, then duplicated it until I had 5 of them. The report seems to load just fine for me, but the query is about as simple and calm as could be, so maybe it's just not using enough juice to encounter the issue.

    SELECT TOP 1 n.Caption FROM Orion.Nodes AS n

  • yeah I would up the timeouts, I would also make sure to space it apart from any other reports you have scheduled, so if you have several that run at 8am have this run at 7am