Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

How did your Orion 2018.4 Upgrade go?

Hello Thwackers,

How did everybody go with the upgrade to Orion 2018.4?

From our experience, I would say it's the worst upgrade ever, and I have been managing/upgrading Orion since 2008. We have about 30 polling engines deployed globally in our data centers which HA configured on each. The actual installation and upgrade went fine, or so we thought. Several hours after the upgrade, we start seeing alerts in the web console for 'node not polled in the last 10 minutes'. Upon further investigation, we see that four of our polling engines have stopped polling all nodes they monitor, checking the servers, all the services are running, the polling engines are showing up in the web console, there's nothing to indicate something is wrong. After working with support, sending diagnostic files back and forth we are still no closer to fixing this after a month of working on it. We even completely rebuilt one of the affected polling engines by deploying a clean Windows 2016 VM and installing SolarWinds fresh on it, worked for a few hours then the same problem started again.

Has anyone else had any similar issues with their additional polling engines since upgrading to Orion 2018.4?

Find more posts tagged with

Accepted answers

All comments

superfly

Yeah I had the same issue. My main poller stopped polling devices. I ended up running the configuration wizard again which fixed the issue for us.

I also noticed that when adding new widgets, only 2 would show up. After re-running the config wizard, all the widget option showed up again. Since then when I've had to re-run the config wizard, the widgets options can return to just showing 2 meaning I have to re-run the wizard again.

Hope you get it sorted.

dgsmith80

Hi b3nw00d I would be interested to hear about your progress with support. Did you find the issue or is it still ongoing?

superfly99 I think that might be attributed to the IIS, I suspect the 2 widgets your getting are PerfStack Projects. If you rebuild your IIS you might see it improve. It's also worth checking it's not a browser issue.

superfly

I can't remember which 2 widgets it showed but it was certainly not a browser issue. Happened on multiple machines with either IE or Chrome.

The upgrade brought about a few issues which fortunately we were able to iron out. But we still need to keep an eye on the widgets if we have to re-run the config wizard.

b3nw00d

Hi dgsmith80, support have raised the issue with development, they seem to think that the problem is related to the fixes HF2 is supposed to resolve. We see some of the SolarWinds services consuming large amounts of memory and high CPU usage, so we are still waiting for feedback from dev

lynchnigel

We to are seeing alerts in the web console for 'node not polled in the last 10 minutes' which has been happening since the upgrade. The rest didnt go too well with changes in the database which caused issues with our DBA guy but they have since been resolved and just the above outstanding - which means a reboot now and again which is a pain.

kasaff

I'm really sorry to hear you're still having trouble after the upgrade! We have not had wide-spread reports of this post hotfix 2, so we'll have to do some more investigation. Could I get you to get some memory dumps when the memory is spiking up and send them into support? That should help us figure out the root cause and get you working as fast as possible. There are directions in this KB article:

Capture a memory dump - SolarWinds Worldwide, LLC. Help and Support

Thanks very much!

b3nw00d

Sure, I can do that. I restarted the servers that were not polling a few days ago and surprisingly they are still polling everything, still seeing high memory usage on a couple of servers. This is what I see on a couple of our servers

solaradmin

Have you gotten this resolved?

wluther

Orion Platform 2018.4 HF3, SCM 1.1.0, NCM 7.9, NPM 12.4, DPAIM 11.1.1, VMAN 8.4.0, SAM 6.8.0, NetPath 1.1.4

We upgraded to latest release mid March, and things were going fairly well for nearly a week. We have had to restart the services on our polling engines due to "random" data gaps, when the polling engines had stopped polling/collecting data from everything on the specific poller. A day or so after resetting services on poller 3, then poller 2 did the same thing. Last sync/minutes since last sync/next poll times/etc. all show times stuck in the past, when the pollers stopped polling (obviously). The MSMQ counters also start going up in correspondence. (On a side note, I thought these queues were supposed to hold the data that couldn't make it into the db, then push it through once it could. Yet, I don't believe this has ever happened. We ALWAYS just have a data gap, which probably means I just simply do not properly understand their purpose.)

Support continues to tell us it's due to our pollers failing to connect to the DB, which is our problem. Just as soon as I restart the "Orion Module Engine" service, however, everything seems to start working again. During the time where the polling engine is not collecting data, NOTHING within our Orion environment shows to actually be having a problem. If I wasn't explicitly looking for this issue, or hadn't been using the last poll/sync alert, then I would think all was well, and I have tons of historical data to view. The first few times this happened, having maybe a month or so of time in between, we accidentally found out when someone was running their monthly report, and there were no bandwidth/interface metrics. Nothing turned red, yellow, grey... Everything was green and appeared to be working just fine.

We have actually seen this similar behavior for approximately 1 1/2 years now, off and on, across Windows/SQL Servers 2012, r2, & 2016, as well as all the Orion/NPM/NCM/SAM/DPA/etc. versions in between. I have given them (support) the benefit of the doubt for a long while now, since there are in fact a few things we *COULD* do better in our virtual environment. Luckily, we are in the process of building out a couple of new data centers, so support is running out of options for the blame game.

Has anyone else experienced, or is currently experiencing, something similar? zackm m_roberts

Thank you,

-Will

hpstech

I feel your pain. I explain a similar situation in the HA forum.

We had to uninstall and re-install all services to get back on track.

When the Orion template component 'Job Scheduler v2: Results Notified Error' counter is building, this is when stats collection stops, and my stomach turns into knots. And we have to bounce services.

m_roberts

This is always a difficult one, as the SQL connectivity errors you are seeing in the logs are not always truly related to connectivity to the SQL Server. In this circumstance, if you can identify any sort of pattern get additional monitoring in place during this time; Wireshark with filter to the SQL server on the polling engine, SAM monitoring connectivity and KPI's on connections etc.

SW Support sometimes need to be provided evidence to that fact it is not 'environmental' to allow the case to be worked on fully and escalated when necessary.