21 Replies Latest reply on Aug 31, 2018 9:37 AM by cobrien

EOC 2.0 report preview crashes all Orion instances

cory.cousins Jan 30, 2018 8:38 AM

Case#00042056. Latest and greatest version of all modules. Loading a report preview (not a fully fledged report... which never finishes anyway) for SAM metrics within EOC against 3 separate Orion environments spins for about 20-30 minutes before all connected Orion instances become unresponsive, throw out errors, and/or are missing menus. Orion instances only recoverable via reboot of main polling engines. EOC is also only recoverable following reboot. Can be reproduced every time. All info provided to support. Just a caution...

Re: EOC 2.0 report preview crashes all Orion instances

jblankjblank Jan 30, 2018 9:24 AM (in response to cory.cousins)

cory.cousins - Thank you for sharing. What version of EOC are you running in this scenario? Just to clarify, the report you are attempting to run is against a set of servers and a particular performance counter at 1hr intervals for a 24hr time period? You state that this report runs without issue on the local instance, but when running the report from EOC, you have experienced the issue you have mentioned?

I assume you are running an older version of EOC as you indicated you were unable to find diagnostics. In EOC 2.0, the Diagnostics are captured in the exact same manner as your remote instances.
Like (0)

Actions
- Re: EOC 2.0 report preview crashes all Orion instances
  
  cory.cousins Jan 30, 2018 10:57 AM (in response to jblankjblank)
  
  We're running EOC 2.0, which was installed on Jan 4th, 2018 through the falcon installer downloaded on the same day. I downloaded the latest falcon installer this morning (to avoid known issues with older falcon installers) and checked for updates. To my surprise, HF4 was available for Orion Platform 2017.3. Applied the HF, but it didn't solve anything. EOC still crashes our Orion environments on demand.
  
  Like (0)
  
  Actions
  - Re: EOC 2.0 report preview crashes all Orion instances
    
    jblankjblank Jan 30, 2018 11:06 AM (in response to cory.cousins)
    
    With EOC 2.0, the diagnostics are able to be captured in the same manner a remote instance is. Please be sure to upload those to the case. By crashing on demand, you mean when executing the report you attempted to run previously, correct? And to clarify further, this issue is only occurring if attempting to "preview" the report after making changes? It is not occurring by simply running the report?
    
    Like (0)
    
    Actions
    - Re: EOC 2.0 report preview crashes all Orion instances
      
      cory.cousins Jan 30, 2018 11:49 AM (in response to jblankjblank)
      
      The diagnostics will be uploaded as soon as they finish. Yes, this only happens during the preview phase while editing/creating the report. If you simply run a finished report, it just runs for hours and hours and never returns a single result, but at least it doesn't crash the connected Orion instances.
      
      Like (0)
      
      Actions
    - Re: EOC 2.0 report preview crashes all Orion instances
      
      cory.cousins Jul 12, 2018 10:48 AM (in response to jblankjblank)
      
      Upgraded to EOC 2.1 this morning. Now staring at at this error message or similar messages when attempting to access various sections of the web console. On hold waiting for a support resource to troubleshoot.
      
      Like (0)
      
      Actions
      - Re: EOC 2.0 report preview crashes all Orion instances
        
        jblankjblank Jul 12, 2018 11:05 AM (in response to cory.cousins)
        
        Can you clarify which specific views you are seeing this error message. And if there are other views you are seeing a slightly different error message, please also clarify that information as well. Also provide the case number that is opened with support.
        
        Like (0)
        
        Actions
        
        Re: EOC 2.0 report preview crashes all Orion instances
        
        cory.cousins Jul 12, 2018 11:20 AM (in response to jblankjblank)
        
        Happens when you go to home page. There are no menus, but you can use the search function to navigate to a working page where the menus appear again.
        
        All of the menus lead to the error message above... reports, settings, etc.
        
        Been on hold for 55 minutes now waiting for a resource. Hanging up to open a case via customer portal when I have more time.
        
        Like (0)
        
        Actions
        
        Re: EOC 2.0 report preview crashes all Orion instances
        
        jblankjblank Jul 13, 2018 5:33 PM (in response to cory.cousins)
        
        cory.cousins and rfackrell - It sounds like we were able to identify a bug in the RC and were able to provide a workaround for this issue. Please let me know if you see any other issues. We should be able to address this prior to GA. I appreciate both of you sharing your findings.
        
        Like (0)
        
        Actions
        
        Re: EOC 2.0 report preview crashes all Orion instances
        
        rfackrell Jul 14, 2018 10:33 AM (in response to jblankjblank)
        
        Yup.
        So just for the general info and for anyone who might find it useful...
        The Developer I worked with had me log in as the 'admin' account instead of our AD Account, and It loaded just fine.
        He then had me delete the follow from the 'View Types' Section of the EOC Webiste config file (default)under inetpub->SolarWinds->EOC->eoc.config :
        
        <type name="EOCSummary" pagePath="/Orion/SummaryView.aspx" userProperty="SummaryViewID"
        description="@{R=Core.Strings;K=CFGDATA_TM0_25;E=xml}" friendlyName="@{R=Core.Strings;K=CFGDATA_TM0_24;E=xml}" />
        
        then restart IIS, It is now working.
        
        Got to Say jblankjblank, I'm really liking the revisions. Especially the separated resources, and the ability to customize the enterprise summary. I like the last enterprise Summary, but it was a lot of change. And when the 24/7 NOC is used to seeing something one way, the don't particularly like seeing it change.
        I really think this will help get general adoption rates up, and let me release this into our production a lot quicker, and with less questions/grumbling.
        
        1 of 1 people found this helpful
        
        Like (0)
        
        Actions
      - Re: EOC 2.0 report preview crashes all Orion instances
        
        rfackrell Jul 12, 2018 11:58 AM (in response to cory.cousins)
        
        I just did the upgrade, All went well, no errors. In total, took about an hour. However I have the same issue as Cory.Cousins. Unexpected Website Error - The Settings Property 'EOC.SummaryViewID' was not found.
        If I click 'Return to Home' or the Link in the top left, I get this error.
        I can not get to any of the EOC Screens, including the settings, or the reports.I've also tried this on the local server, on via my own machine.
        
        Like (0)
        
        Actions
        
        Re: EOC 2.0 report preview crashes all Orion instances
        
        jblankjblank Jul 12, 2018 3:08 PM (in response to rfackrell)
        
        rfackrell - generate a support case please and upload diags. Then of course post your support case here.
        
        Like (0)
        
        Actions
        
        Re: EOC 2.0 report preview crashes all Orion instances
        
        rfackrell Jul 12, 2018 4:23 PM (in response to jblankjblank)
        
        Case # 00141619
        Just Pending a LeapFile to submit the Diagnostics.
        Thanks!
        
        Like (0)
        
        Actions
Re: EOC 2.0 report preview crashes all Orion instances

rfackrell Mar 26, 2018 1:53 PM (in response to cory.cousins)

Don't have any answers, but following for info.
We have a very similar issue. 3 Regional Pollers, one EOC. We ran a report from the EOC that never came back (No Time out, not error, just spinning) and about 2 hours later, every regional poller crashed. We found that the IIS v3 Service consumed all memory and crashed the box.
We have been unable to reproduce the issue since then. Although this make us really hesitant to move to production with the v2.0 EOC. :/
Like (0)

Actions
- Re: EOC 2.0 report preview crashes all Orion instances
  
  jblankjblank Mar 26, 2018 2:01 PM (in response to rfackrell)
  
  Did you happen to grab diagnostics? From Cory's case, we have been able to make some drastic improvements. If you can share those, it would be highly valuable for the team to investigate.
  
  Like (0)
  
  Actions
  - Re: EOC 2.0 report preview crashes all Orion instances
    
    rfackrell Mar 26, 2018 6:08 PM (in response to jblankjblank)
    
    I Did actually.
    We submitted case 46943, and if diagnostic logs are linked to the Ticket on your end, There should be two sets. The original Diagnostic logs from the day of the crash, and a expanded set with Debugging turned on.
    
    Feel free to PM me if you need any specifics.
    I'd love to help as much as I can.
    Thanks!
    
    Like (0)
    
    Actions
Re: EOC 2.0 report preview crashes all Orion instances

jblankjblank Jul 11, 2018 7:42 PM (in response to cory.cousins)

cory.cousins and rfackrell - I wanted to circle back on this thread and ask if either of you have installed and ran into any issues through the EOC 2.1 RC? Based on both of your feedback and diagnostics, we found areas in which we were able to make drastic improvements. Significant changes have been implemented on the back-end to improve reporting and I am curious on both of your thoughts for this RC.
1 of 1 people found this helpful
Like (0)

Actions
- Re: EOC 2.0 report preview crashes all Orion instances
  
  rfackrell Jul 11, 2018 7:45 PM (in response to jblankjblank)
  
  That's exciting to hear! I'm actually scheduled to put EOC 2.1 into QA This Friday. I'll let you know how it goes.
  
  Like (1)
  
  Actions
  - Re: EOC 2.0 report preview crashes all Orion instances
    
    cobrien Aug 31, 2018 9:37 AM (in response to rfackrell)
    
    Did EOC 2.1 resolve the problem for you?
    
    Like (0)
    
    Actions
Re: EOC 2.0 report preview crashes all Orion instances

cory.cousins Aug 2, 2018 12:31 PM (in response to cory.cousins)

When will EOC match what Orion is able to do from a reporting perspective? In particular, asset inventory.
Like (0)

Actions
Re: EOC 2.0 report preview crashes all Orion instances

cory.cousins Aug 30, 2018 2:51 PM (in response to cory.cousins)

EOC 2.1 still has the same reporting issue that crashes connected Orion environments. We're dropping maintenance on this product at the end of our current renewal cycle.
Like (0)

Actions
- Re: EOC 2.0 report preview crashes all Orion instances
  
  cobrien Aug 31, 2018 9:30 AM (in response to cory.cousins)
  
  Hey cory.cousins, I'm sorry for the hassle. This shouldn't be happening. This is a severe issue but it is happening only in specific environments. It looks like we're having trouble reproducing it in our lab. That always makes it more complex to fix. We implemented a number of fixes (CORE-10393, EOC-1402, EOC-1409, and EOC-1486) in EOC 2.1 which we thought fully addressed the issue. In your specific case, it's definitely not.
  
  It looks like the support and Dev team have been working with you and making progress, but it's slow going. Your help is particularly valuable because we only have a couple of users experiencing this issue. I understand if you no longer want to invest your time to continue that effort. We'll keep trying.
  
  Like (0)
  
  Actions

Go to original post