cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 10

EOC 2.0 report preview crashes all Orion instances

Case#00042056.  Latest and greatest version of all modules.  Loading a report preview (not a fully fledged report... which never finishes anyway) for SAM metrics within EOC against 3 separate Orion environments spins for about 20-30 minutes before all connected Orion instances become unresponsive, throw out errors, and/or are missing menus.  Orion instances only recoverable via reboot of main polling engines.  EOC is also only recoverable following reboot.  Can be reproduced every time.  All info provided to support.  Just a caution...

0 Kudos
21 Replies
Level 10

EOC 2.1 still has the same reporting issue that crashes connected Orion environments.  We're dropping maintenance on this product at the end of our current renewal cycle.

0 Kudos

Hey cory.cousins​, I'm sorry for the hassle.  This shouldn't be happening.  This is a severe issue but it is happening only in specific environments.  It looks like we're having trouble reproducing it in our lab.  That always makes it more complex to fix.  We implemented a number of fixes (CORE-10393, EOC-1402, EOC-1409, and EOC-1486) in EOC 2.1 which we thought fully addressed the issue.  In your specific case, it's definitely not.

It looks like the support and Dev team have been working with you and making progress, but it's slow going.  Your help is particularly valuable because we only have a couple of users experiencing this issue.  I understand if you no longer want to invest your time to continue that effort.  We'll keep trying.

0 Kudos
Level 10

When will EOC match what Orion is able to do from a reporting perspective?  In particular, asset inventory.

pastedImage_0.png

0 Kudos
Product Manager
Product Manager

cory.cousins​ and rfackrell - I wanted to circle back on this thread and ask if either of you have installed and ran into any issues through the EOC 2.1 RC?  Based on both of your feedback and diagnostics, we found areas in which we were able to make drastic improvements.  Significant changes have been implemented on the back-end to improve reporting and I am curious on both of your thoughts for this RC.   

That's exciting to hear! I'm actually scheduled to put EOC 2.1 into QA This Friday. I'll let you know how it goes.

Did EOC 2.1 resolve the problem for you?

0 Kudos
Level 10

Don't have any answers, but following for info.

We have a very similar issue. 3 Regional Pollers, one EOC. We ran a report from the EOC that never came back (No Time out, not error, just spinning) and about 2 hours later, every regional poller crashed. We found that the IIS v3 Service consumed all memory and crashed the box.

We have been unable to reproduce the issue since then. Although this make us really hesitant to move to production with the v2.0 EOC. 😕

0 Kudos

Did you happen to grab diagnostics?  From Cory's case, we have been able to make some drastic improvements.  If you can share those, it would be highly valuable for the team to investigate. 

0 Kudos

I Did actually.
We submitted case 46943, and if diagnostic logs are linked to the Ticket on your end, There should be two sets. The original Diagnostic logs from the day of the crash, and a expanded set with Debugging turned on.

Feel free to PM me if you need any specifics.

I'd love to help as much as I can.

Thanks!

0 Kudos
Product Manager
Product Manager

cory.cousins - Thank you for sharing.  What version of EOC are you running in this scenario?  Just to clarify, the report you are attempting to run is against a set of servers and a particular performance counter at 1hr intervals for a 24hr time period?  You state that this report runs without issue on the local instance, but when running the report from EOC, you have experienced the issue you have mentioned?

I assume you are running an older version of EOC as you indicated you were unable to find diagnostics.  In EOC 2.0, the Diagnostics are captured in the exact same manner as your remote instances.   

0 Kudos

We're running EOC 2.0, which was installed on Jan 4th, 2018 through the falcon installer downloaded on the same day.  I downloaded the latest falcon installer this morning (to avoid known issues with older falcon installers) and checked for updates.  To my surprise, HF4 was available for Orion Platform 2017.3.  Applied the HF, but it didn't solve anything.  EOC still crashes our Orion environments on demand.

0 Kudos

With EOC 2.0, the diagnostics are able to be captured in the same manner a remote instance is.  Please be sure to upload those to the case.  By crashing on demand, you mean when executing the report you attempted to run previously, correct?  And to clarify further, this issue is only occurring if attempting to "preview" the report after making changes?  It is not occurring by simply running the report?

0 Kudos

Upgraded to EOC 2.1 this morning.  Now staring at at this error message or similar messages when attempting to access various sections of the web console.  On hold waiting for a support resource to troubleshoot.

pastedImage_1.png

0 Kudos

I just did the upgrade, All went well, no errors. In total, took about an hour. However I have the same issue as Cory.Cousins. Unexpected Website Error - The Settings Property 'EOC.SummaryViewID' was not found.

If I click 'Return to Home' or the Link in the top left, I get this error.
I can not get to any of the EOC Screens, including the settings, or the reports.I've also tried this on the local server, on via my own machine.

0 Kudos

rfackrell​ - generate a support case please and upload diags.  Then of course post your support case here. 

0 Kudos

Case # 00141619
Just Pending a LeapFile to submit the Diagnostics.
​Thanks!

0 Kudos

Can you clarify which specific views you are seeing this error message.  And if there are other views you are seeing a slightly different error message, please also clarify that information as well.  Also provide the case number that is opened with support. 

0 Kudos

Happens when you go to home page.  There are no menus, but you can use the search function to navigate to a working page where the menus appear again. 

pastedImage_0.png

pastedImage_1.png

All of the menus lead to the error message above... reports, settings, etc.

Been on hold for 55 minutes now waiting for a resource.  Hanging up to open a case via customer portal when I have more time.

0 Kudos

cory.cousins and rfackrell - It sounds like we were able to identify a bug in the RC and were able to provide a workaround for this issue.  Please let me know if you see any other issues.  We should be able to address this prior to GA.  I appreciate both of you sharing your findings. 

0 Kudos

Yup.
So just for the general info and for anyone who might find it useful...
The Developer I worked with had me log in as the 'admin' account instead of our AD Account, and It loaded just fine.
He then had me delete the follow from the 'View Types' Section of the EOC Webiste config file (default)under inetpub->SolarWinds->EOC->eoc.config :


    <type name="EOCSummary" pagePath="/Orion/SummaryView.aspx" userProperty="SummaryViewID"

          description="@{R=Core.Strings;K=CFGDATA_TM0_25;E=xml}" friendlyName="@{R=Core.Strings;K=CFGDATA_TM0_24;E=xml}" /> 


then restart IIS, It is now working.

Got to Say jblankjblank​, I'm really liking the revisions. Especially the separated resources, and the ability to customize the enterprise summary. I like the last enterprise Summary, but it was a lot of change. And when the 24/7 NOC is used to seeing something one way, the don't particularly like seeing it change.
I really think this will help get general adoption rates up, and let me release this into our production a lot quicker, and with less questions/grumbling.