This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.

You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Solarwinds Performance is Horrible after 12.3 upgrade

FormerMember over 5 years ago

I am wondering why the discussion Solarwinds is now horribly unstable. is locked? For an ongoing issue that we also are experiencing, why would thwack moderators choose to lock the discussion and prevent other customers from weighing in, and asking questions? I am attempting to apply the work around in the post, but I wanted to open another thread since the other one is locked. Hopefuly other customers that have upgraded to 12.3 can communicate on thwack. I found that the memory of the Cortex service was using 24-29GB of the 32GB available on the server of my primary poller, and it causes the web console and polling to not function. I'll reply back to this thread after I apply the proposed work-around on the other thread linked above.

OUR ENVIRONMENT: Primay Poller (HA Primary), 18 Additional Pollers (9 Active / 9 HA), 2 AWS, Physical SQL Server with 32 cores, 148 GB Memory, Raid 10 LUNs on 3 Par with SSD.

Network Elements

TOTAL COUNT	81350
NODES	5264
INTERFACES	51321
VOLUMES	24765

Top Replies

0 neomatrix1217 over 5 years ago

Have you installed the hotfix that was released today?
Orion Platform 2018.2 Hotfix 3 - SolarWinds Worldwide, LLC. Help and Support
Cancel
Vote Up +2 Vote Down

Cancel
0 FormerMember over 5 years ago in reply to neomatrix1217

No, I'll look at it now. Thank You for bringing that to my attention.
Cancel
Vote Up 0 Vote Down

Cancel
0 daklein over 5 years ago in reply to neomatrix1217

It'd be really great if there were a note as to what's different between HF2 (which I had to back out of due to mass install & config failures) and the new HF3.
I've done all the Buffered-not-Streamed settings and TCP reghacks, AND doubled our main poller's specs, yet we're at slightly less than a max of two days' uptime before we need to restart all services on our mail poller and two satellite pollers.
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 5 years ago in reply to neomatrix1217

HF3 was no better, we have a ticket open with SW Support, no word yet. They have the diags, it is something that brings down the entire environment as it overloads the Database. I have notice that service sending the queries that cause the waits and later the blockers, is the Cortex service. Our DBA created an index for the Events table, and that seems to have given stability until SW can come up with a fix.
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 5 years ago in reply to daklein

I have applied the Buffered as well as HF3, no improvement.
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 5 years ago

Update: Solarwinds Support/Engineering Ticket still does not have a fix for this issue. The ticket has been open since July 6th, and they have been evaluating our diagnostics since that week.

Support Ticket:
00137887
Cancel
Vote Up 0 Vote Down

Cancel

0 aLTeReGo over 5 years ago in reply to FormerMember

msawyer wrote:

Update: Solarwinds Support/Engineering Ticket still does not have a fix for this issue. The ticket has been open since July 6th, and they have been evaluating our diagnostics since that week.

Support Ticket:

00137887

We're sorry to hear that you're still experiencing issues, even after applying Hotfix 3. kasaff is working with support now to have your case escalated.

0 dgsmith80 over 5 years ago in reply to daklein

Check out this link: Orion Platform 2018.2 Hotfix 3 - SolarWinds Worldwide, LLC. Help and Support it lists what has been addressed by each of the HotFix's
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 5 years ago in reply to aLTeReGo

Update: We received a hotfix about 2 weeks ago and our environment has remained stable since that was put in place. The fix was to replace an app.settings.json file that changed the behavior of the cortex service and how it handles data. Support informed me that they would include in it an upcoming release or hotfix.
Cancel
Vote Up +1 Vote Down

Cancel
0 mattyc88 over 5 years ago in reply to FormerMember

Is this still in a stable condition?
We've having major issues with the Cortex service and have been since HF1 (Currently got HF3 applied)
On our primary poller the Cortex service is using 31GB/48GB, all the APEs have a very similar ratio.
Cancel
Vote Up 0 Vote Down

Cancel