Need help with hanging reports

Hi everyone,

I'm looking for a collective solution to our issue as all Solarwinds Support can do is ask for diagnostic reports...., again and again!

Here's the breakdown

  • We have a ton of custom web-based reports with multiple dynamic query builder queries.  It's also worth noting that we have a good number of web-based reports with multiple SWQL queries in them.
  • We recently upgraded from Solarwinds build 2023.1 RC1 to general release Solarwinds build 2023.1.
  • Now all reports with multiple dynamic query builder queries or multiple SWQL queries just hang when we open them.
  • I'm able to reproduce this issue by building a super simple report.
    • The report opens successfully when I limit the number of queries to 4 or less.
    • The report hangs and won't open when I hit the 5 or more mark.
      • It's worth nothing these can be 5 different data-source queries or 5 of the same data-source queries.
      • It's also worth nothing that the report hangs no matter if all 5 queries are the same or different.
    • I'm able to successfully open the report once I delete the queries back down to 4 or less.
  • Here's what I see in the OrionWeb.log when the report hangs:
    • 2023-03-31 13:34:27,049 [312] (71) WARN  ASP.global_asax - (null)  Long request time: [url:https://10.132.163.34/Orion/Report.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4][time:7683]

      2023-03-31 13:36:23,811 [308] (77) WARN  ASP.global_asax - (null)  Long request time: [url:https://10.132.163.34/Orion/Reports/Preview.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4&waitid=0744d160ee3f4196ae99bd8558376df2][time:5921]

      *** Assembly App_Web_nodemanagement.asmx.43409ddd, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly App_Web_nodemanagement.asmx.193e71da, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.Orion.Discovery.Contract, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.Orion.Core.SharedCredentials, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly App_Web_nodeproperties.ascx.891fea64, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.UDT.Common, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      *** Assembly SolarWinds.SysMan.Utils, Version=2023.1.0.0, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

      2023-03-31 14:00:53,843 [306] (79) WARN  ASP.global_asax - (null)  Long request time: [url:https://10.132.163.34/Orion/Reports/Preview.aspx?ReportID=2564&ReturnTo=aHR0cHM6Ly8xMC4xMzIuMTYzLjM0L09yaW9uL1JlcG9ydHMvRGVmYXVsdC5hc3B4&waitid=0744d160ee3f4196ae99bd8558376df2][time:19456]

  • I found several Solarwinds KB articles that talked about ASP.global_asax warning messages, but non of them really cover the same issue as we're experiencing.
  • Again I asked SW support for their option based on my findings and all I got was "please send another diagnostic report" . 

At this point I feel I can get better support from you all on Thwack, Reddit or a random homeless person on the street than from Solarwinds support!!!!

Now I know the routine where someone from SW jumps on says "case # please", so before you ask it's #01311386.

I also know there's a tendency to censor THWACK posts so, I'm going to post this on the Reddit /Solarwinds.

Parents
  • I'd be curious if you put your 5 queries in a modern dashboard (each as a separate resource) do you get the same issue? If so it may indicate an issue with the backend of SWIS/SWQL queries in some odd limitations. If not its likely just something in the report engine, unfortunately long request times are hard to track down if its actual performance or not. If the modern dashboard loads fine you could try to up timeouts on your reports to see if that affects it. 

  • Jeremy, great question!  I just checked and there are no issues with the modern dashboards that I've built. I have MD with 12 different queries that opens nice and quick with no issues.

    Yes this one is a bugger!!

    Here's what I have for Report timeouts.

     

    For SWNetPerfMon.db it's the default 90:

    ! Database Command timeout in seconds

    CommandTimeout=90

    and web.config is default exec time off 600 for everything except the schedule which is 300

    <location path="Orion/ReportWait.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="600" />
    </system.web>

    <location path="Orion/Report.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="600" />
    </system.web>

    <location path="Orion/Reports/AsExcel.aspx">
    <system.web>
    <httpRuntime maxRequestLength="64384" executionTimeout="600" />
    </system.web>

    <location path="Orion/Reports/Preview.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="600" />
    </system.web>

    <location path="Orion/Reports/Scheduler.aspx">
    <system.web>
    <httpRuntime maxRequestLength="16096" executionTimeout="300" />
    </system.web>

  • You could try to up them again, the article they have for this has you set those values to 3600, outside of that I'm not on 2023 to be able to see if I can repro it. I'd keep pushing in support to see if you can get to one of the good techs. 

    support.solarwinds.com/.../Large-report-error-Unexpected-Website-Error-Request-timed-out

  • Coffee is still kicking in just had a workaround idea. If these queries load fine for a modern dashboard can you just set your report to run using the URL for that dashboard? 

  • I'm sorry I don't quite understand what your suggesting.  Are you suggesting rebuilding the reports as modern dashboards? If so, that would be decent work around option however most users like to have their reports ran on a scheduled bases and mailed.  Also it doesn't appears there's a PDF or Excel export function in MD yet.

Reply Children