EOC 2.0 report preview crashes all Orion instances

Case#00042056.  Latest and greatest version of all modules.  Loading a report preview (not a fully fledged report... which never finishes anyway) for SAM metrics within EOC against 3 separate Orion environments spins for about 20-30 minutes before all connected Orion instances become unresponsive, throw out errors, and/or are missing menus.  Orion instances only recoverable via reboot of main polling engines.  EOC is also only recoverable following reboot.  Can be reproduced every time.  All info provided to support.  Just a caution...

  • cory.cousins - Thank you for sharing.  What version of EOC are you running in this scenario?  Just to clarify, the report you are attempting to run is against a set of servers and a particular performance counter at 1hr intervals for a 24hr time period?  You state that this report runs without issue on the local instance, but when running the report from EOC, you have experienced the issue you have mentioned?

    I assume you are running an older version of EOC as you indicated you were unable to find diagnostics.  In EOC 2.0, the Diagnostics are captured in the exact same manner as your remote instances.   

  • We're running EOC 2.0, which was installed on Jan 4th, 2018 through the falcon installer downloaded on the same day.  I downloaded the latest falcon installer this morning (to avoid known issues with older falcon installers) and checked for updates.  To my surprise, HF4 was available for Orion Platform 2017.3.  Applied the HF, but it didn't solve anything.  EOC still crashes our Orion environments on demand.

  • With EOC 2.0, the diagnostics are able to be captured in the same manner a remote instance is.  Please be sure to upload those to the case.  By crashing on demand, you mean when executing the report you attempted to run previously, correct?  And to clarify further, this issue is only occurring if attempting to "preview" the report after making changes?  It is not occurring by simply running the report?

  • The diagnostics will be uploaded as soon as they finish.  Yes, this only happens during the preview phase while editing/creating the report.  If you simply run a finished report, it just runs for hours and hours and never returns a single result, but at least it doesn't crash the connected Orion instances.

  • Don't have any answers, but following for info.

    We have a very similar issue. 3 Regional Pollers, one EOC. We ran a report from the EOC that never came back (No Time out, not error, just spinning) and about 2 hours later, every regional poller crashed. We found that the IIS v3 Service consumed all memory and crashed the box.

    We have been unable to reproduce the issue since then. Although this make us really hesitant to move to production with the v2.0 EOC. :/

  • Did you happen to grab diagnostics?  From Cory's case, we have been able to make some drastic improvements.  If you can share those, it would be highly valuable for the team to investigate. 

  • I Did actually.
    We submitted case 46943, and if diagnostic logs are linked to the Ticket on your end, There should be two sets. The original Diagnostic logs from the day of the crash, and a expanded set with Debugging turned on.

    Feel free to PM me if you need any specifics.

    I'd love to help as much as I can.


  • cory.cousins​ and rfackrell - I wanted to circle back on this thread and ask if either of you have installed and ran into any issues through the EOC 2.1 RC?  Based on both of your feedback and diagnostics, we found areas in which we were able to make drastic improvements.  Significant changes have been implemented on the back-end to improve reporting and I am curious on both of your thoughts for this RC.   

  • That's exciting to hear! I'm actually scheduled to put EOC 2.1 into QA This Friday. I'll let you know how it goes.

  • Upgraded to EOC 2.1 this morning.  Now staring at at this error message or similar messages when attempting to access various sections of the web console.  On hold waiting for a support resource to troubleshoot.