Anyone have any upgrade experiences for the general release?
We are looking to upgrade our entire SW suite in addition to NPM and would like to hear any feedback.
Also, is the RC code the same as the GR code?
Thanks in advance
This thread has been super helpful and given us many conversation points and feedback. However, the thread is now gotten unwieldy and hard to parse through the feedback. I'm locking this thread and for those of you who are participating in our RCs, invite you to come give additional feedback on my post Orion Platform: Preparing for the Upgrade to 2018.2 in the NPM release candidate forum. Once we are out of RC, I'll be starting a new topic thread for upgrade feedback for NPM 12.3.
Thank you and for those of you with whom I have ongoing conversations, we'll find other avenues to move forward on those topics.
What the Version now ?
12.1
Everything went extremely smooth for me. I upgraded NPM, SAM, NCM, UDT, IPAM, and VIM. Much faster upgrade than normal and didn't run into any issues.
We built out a new server to finally get off of 2008. I was rather dreading this migration and update but it could not have been smoother. I scheduled a 6 hour downtime window and we were back up and running in just over two ( and of course half of that time was waiting for the Website Optimization to complete ) We migrated/updated NPM, NCM, IPAM, SAM, WPM and NTA. No issues.
Thanks. So you built a new box(s) and pointed to old DB?
I’m doing in place. And support is giving me wildly conflicting upgrade plans since the PUA has the known bugs.
Interesting. I'm a version back--NPM 12.0.1. The PUA says it can't handle the upgrade to 12.2. Some internal SW folks told me in a Thwack meeting today that I had to download another set of app installers to get the job done. Then Support told me to just use the PUA to get to 12.1, then use the NEW PUA to go to 12.2.
Sadly I'm dealing with Server 2008 issues, so until I get the new VM licensing for 2012 resolved, I'm on hold for 12.2.
Yes, built out a new server and brought it up using the name/ip of our older server. Installed software and updated all in one step using our old DB. As worried as I was about this one it turned out to be the smoothest SW upgrade I have ever done.Roughly followed this upgrade doc
I brought a server on NPM 12.0.1, SAM 6.3.0, NCM 7.5.1, UDT 3.2.4, and SRM 6.3.0 directly to the current releases with the new installer without any issues. There is no need to take an interim step of going to 12.1. One word of warning, the installer will check for incompatibilities with existing modules and will refuse to proceed if any are found. In this case, the client had installed a version of the Patch Manager Orion module that was not compatible with their other existing modules and the new installer wouldn't proceed with it installed. Once the PM module was uninstalled, the upgrade went through without issue.
Upgraded from NPM 11.5.2, NTA, SAM, and SRM for that version all the way to 12.2. took a while but the Upgrade advisor helped a lot. only issue is that every time a page refreshes the stupid pencil popup screen keeps coming up. how do you stop that? this is driving me crazy and I can not figure out how to turn it off.
serena can correct me if I'm mistaken, but simply clicking 'Ok, Got it' should make this notification go away forever for your user account.
We get the same pop up over and over and over as well, and yes we click it but it keeps coming back.
Yes clicking "ok got it" will persist the setting to not show this any more. Clicking away from the popup or anywhere not the link that confirms "ok got it" won't persist it to stay away, it will just dismiss it for this login.
rschroeder wrote:Sadly I'm dealing with Server 2008 issues, so until I get the new VM licensing for 2012 resolved, I'm on hold for 12.2.
rschroeder wrote:
This will put my upgrade on hold as well, except in my case I also have FoE, Pysical servers, and an unsupported database server version :-(
/RjL
Finally an all-in-one update installer!
My upgrade went relatively smooth. I had a little trouble with upgrading my additional poller because it kept timing out. The poller install package has a hard limit of 30 min to gather all the info from the main poller. I found that if you grab 'CoreInstaller.msi' and 'MIbs.msi' from C:\ProgramData\Solarwinds\Installers on the main poller and manually copy them to your additional poller you can cut a bunch of that 30 min window. Copy those files to your desktop. Then start the install on the additional poller. As soon as you start, you need to look for C:\Users\[your user]\AppData\Local\Temp\2\SWOrionSetup[datestamp]\subinstallers\CORE_2017.x\. (This directory won't be created until you start your install.) Immediately paste those files you copied over to your desktop into this directory and it'll shave off almost a gig of download (depending on the number of modules in your environment) and help keep you from timing out.
It's well worth the upgrade. I'm looking forward to future upgrades with the new all-in-one installer.
The only feedback i have is that after the upgrade a number of thresholds for ASA hardware health seemed to be messed up (using Orion defaults). to stop the alerts i had to go in there and set custom thresholds. The values it populated were correct, just the thresholds were messed up.
But im loving the new ASA insight and ACL insight it has.
As far as upgrade process i enjoyed the new installer that they have for it. No longer needing to go through the upgrade adviser was nice. was able to update 4 products (NPM 12.2, NCM 7.7, SAM 6.4 HF 1, and NTA 4.2.3) all in under an hour. And there were no issues or errors with the actual install process. Like i said the only problem that i encountered is what was described above.
Wonderful! thanks for the feedback
Thanks for the feedback! Did you create a support ticket for the issue you described?
This afternoon I had a successful upgrade from NPM 12.0.1 to 12.2. Thanks to serena and her peers, I think we all learned some useful things about users like me, and ASA 5525-X's, and the upgrade process.
Time will tell how much new learning curve I have in front of me, but this upgrade certainly shows tremendous progress towards simplifying and making a "complete" upgrade of all SW products intuitive and successful.
It went more smoothly, and much better, with less frustration on my part, than all previous Solarwinds upgrades I've done.
Kudos to you Solarwinds!
Yay this post make smile
Yes i have an active support case for this. We are currently working through it, we've discovered that when NPM polls the devices it is returning a temp of 0 degrees celsius which is inaccurate. I believe my ticket is being escalated to dev for support.
Environment before upgrade.
NPM 12.1, NTA 4.2.2, VMAN 7.1, IPAM 4.5.0, NCM 7.6, SAM 6.4, SRM 6.4
First about a month ago I started preparing and moved our main NPM/Orion from 2008 server to 2016. Then a few weeks ago moved NTA from 2008 to 2016. Our SQL was/is already on a new platform being supported. We used to have STM or the old profiler running on a 2008 server too, but no longer use it so just disabled that integration to be left with just SRM.
Ran the new install file. I didn't note all the warnings, but nothing was a red flag not allowing me to continue. Even though we have more RAM and CPU cores than required, it complained because our VM environment has 2.2GHz and not the required 3.0 I think it said. Another warning was on our VMAN 7.1 saying we would either need to migrate after the upgrade, or moved to VMAN 8 appliance.
This was probably the smoothest large update I have done as we had 5 product updates. I was nervous and worried. Once finished, I ran NTA 4.3.2 on our separate DB server. I then ran the VMAN wizard to change from 7.1 appliance to the new VMAN Orion poller.
Total outage was around 35 mins for us.
I have noticed some bugs/issues, but as for the upgrade process I'm relieved and hope they continue to build on this and become more comfortable for all.
Can I get some assistance on getting this ticket routed to the correct people? I am currently dealing with a technician that is terrible at communication and have no idea what's going on with my ticket even after numerous attempts over several days requesting an update. My ticket number is 1253744. Below is a screenshot of his latest reply lol
Upgrade 12.2 RC1 - RC3 to 12.2 official. Seems kind of weird, but RC3 became the defacto official.
Anyway, I have noticed some interesting anomalies that have been a bear to troubleshoot.
First: One of my ASA devices would not show serial number and site to site VPN. In fact I have to make sure that only the Node Details my SolarWinds is selected to provide the SNMP data for the details. Anything other poller the device will become an unknown.
Second: I just went and looked at the reports to find the report that provides the inventory which would let me mitigate the fact that one of three ASA devices doesn't show the serial number and behold, there are 0, ZERO reports listed.
Third: One of the best outputs of in these recent versions is being able to monitor site to site VPNs, stopped working. While working with tech support I found out that there is going to be a HOT FIX to repair something.
I am working to get a separate SQL server stood up which will help I think and eventually a physical server to remove it from the sometime sketchy virtual environment.
SolarWinds Orion Platform 2017.3 Hotfix 1 includes the following security updates:
https://downloads.solarwinds.com/solarwinds/Release/HotFix/OrionPlatform-v2017.3.5320-HotFix1.zip
Password fields allowed auto-complete.
The Manage Alerts page displays $ signs instead of variables.
Node limitations are applied incorrectly on tertiary-level hosted entities, such as application components, in container members.
Account limitations to display no objects are not correctly applied when viewing hosted entities in container members.
Borders around popups in bar graphs are bigger than the popup in Chrome.
Cannot add more than one variable in the mail recipients section in the Send an Email/Page alert action.
The bundled SNMP Walk tool is an earlier version.
Installations behind proxy servers may not be able to connect to Leapfile to upload diagnostic files.
Ok, got it dialog does not dismiss correctly.
NOC mode for ASA Node Details page does not load in a timely fashion.
ASA-related charts do not load in the Node Details’ subviews.
Orion Platform does not support later versions of DPAIM.
Toolset menus do not display in the correct area.
The Orion Web Console is unresponsive when hovering over Toolset buttons in the management resource.
The Orion Web Console does not display the top menu and shows errors on other pages when client certificate authentication is enabled in IIS.
so i went with the upgrade just now i have npm, ncm, ipan and udt
and this is the error i got, and now i have a complete unusable system
Can you check in add/remove programs if you have two IPAM's installed.
"I have a complete unusable system" At this point in the installer, nothing has changed on your existing installation. Can you clarify what you mean by "unusable system"?
You can get around this issue by uninstalling IPAM. Don't run the configuration wizard when you remove the product at this stage. Then, you would run the 12.2 installer again. On the first screen, expand "Select Additional Products" to select IPAM. This will install IPAM + upgrade your existing system all in one go, and you'll be up to date on the latest.
Seems to be based on a cookie. Log in on a different machine and you get the pop up again.
This hotfix left me with an unusable system. I have 4 open tickets now and they have been escalated to dev.
The 'ok, got it' is not based on a cookie. This issue is also fixed with hotfix 1 which is available for download through the Customer Portal.
What is your case number?
I get this on mine.The new installer needs some work
If I might be so bold as to offer Installer Error Reports suggestions:
When you get this, which installer are you running? You can get around this error by uninstalling Active Diagnostics and then running the installer again.
Performed the upgrade this afternoon (NPM, UDT, NTA, NCM). Installer went smoothly and probably around an hour. NPM 12.1 was the old platform and Win 2K16 server.
Unfortunately, it created havoc in my environment. My most critical ASA 5520 failover pair became unusable and started rebooting. I had to unmanage them in Orion to get them stable again (after powering off each as Orion was reporting them with dead fans and temp warnings as well as the reboots). It also caused a number of Cisco APs in my environment to become unresponsive and reported them as rebooting repeatedly. The telnet/SSH console was so unresponsive that I couldn't manage to login. Changed them to ICMP only polling and service was restored.
Also have a Citrix SD-WAN appliance with a similar symptom and reporting in Orion as rebooting every 30 minutes now. I checked the appliance and it doesn't appear to be restarting so this may be a SYSUPTIME reporting issue, although I have 4 other SD-WAN appliances on the network that aren't reporting issues...
I have a case open and awaiting feedback.... beware on upgrading your prod environment.
PS: I noticed some else reported the 0C temp reporting on their ASA. I have a 5525-X failover pair reporting the same. It is also reporting the fans are all spinning at 0 rpm. Might need to report that as well.
Cheers,
Aaron
Can you send us your support ticket #?
Where you able to do this with only the 12.2 installer or did you have to do interim steps before it?
Hello,
We've some serious problem after the upgrade from version 12.1 to 12.2.
The installation part was fine without any error but the license store is now corrupted.
We tryed to follow this thread :
https://support.solarwinds.com/Success_Center/Orion_Platform/Orion_Documentation/Corrupted_license_store
However we have an error when trying to launch migration.exe /ExpireAll command , reporting that The license store is not in valid state
Anyone have this problem?
We've already opened the Case #1295467 but we're still awaiting an answer from the support service .
Thanks in advice
I'll have to watch this thread...
we have upgraded the 12.1 version to the latest version 12.2 and installed other modules and from 13 September 2017 CPU utilization is above 95% on main poller and additional polling engines. Dev environment is also showing the same behavior. Core BusinessLayerHost and NCM Businesslayer process eating up around 70% of the CPU. Now Development team asked to for the procdump to check what is the issue. Still Support don't have any definite answer why it is happening.
Keep us posted on this...
so, as i said, nothing working, no NPM/NCM nothing
i've now got it working again, after many hours of work
by uninstalling all the applications, then 1 by one installing them again
what a ballache
Did you use the installer or use the individual installers? I had issues with the 12.1 upgrade using the installer on APEs that required gutting SW from the APEs and installing manually. NTA was the isssue.
The upgrade was smooth, I liked the new monitoring for ASA and I see a great future for that, but I am also having the same issue as I see many people complaining regarding Health Monitoring at some of my ASA's.
I have opened a ticket to check that and the conclusion was that the IOS version of the devices I am having trouble are older than the ones that are showing the information correctly.
OK, I understand that but before the firewall insight feature, it was possible to verify these sensors at that devices. And I cannot simply go ahead and upgrade many devices that are currently running stable only to correct monitoring system problems.
I hope there is a hotfix coming soon.
There IS a hot fix available for NCM 12.2; I don't know if it addresses your ASA issues, though. Check it out and apply it--it sounds like it's helped plenty of people already.
I recently upgraded to NCM 12.2, and applied the hot fix. The only issues I've seen so far have been my ASA 5525-X models aren't getting the right hardware (CPU/Memory/Fan/Temp) polling anymore, and the default views of classes of nodes don't automatically include latency graphs. Those are VERY easy to add into NPM's views, and applying it to one ASA's views applies it to all ASAs views.
There's definitely a learning curve to cover the ways I saw & did things with NPM 12.0.1 and 12.2. But it's been pretty intuitive, and the installers are only getting easier and more comprehensive.
I worked with Solarwinds Technical Support today on a different topic, and learned that a new, more-global upgrade / hot fix installer is being developed and used/tested in-house at SW. It sounds wonderful, and I bet by this time next year we'll all be able to use it to upgrade and patch multiple servers & pollers & Orion applications with a single click.
would you be interested in sending us your diagnostics?
I checked out the Shared Thwack Pollers and found one that provided much more ASA hardware & interface information. I tested it and found it compatible with my ASA's. Downloaded it, applied it, and now my firewall admins are significantly more impressed with NPM.
Check it out, see if it is good for your ASA's.
Thanks Serena. My ticket number is 1300154
Hi Serena,
I have just just found that the "default" view for a device that is not identified correctly is now "ASA View". So all the Access Points I moved to ICMP polling and any not using SNMP is displaying in ASA view which is completely useless for anything other than a ASA firewall.
I might log another ticket to get the default views restored.
EDIT: Found the setting in views and found the "unknown" view had changed to "Node Details ASA Summary" during the upgrade. Changed this back to "Node Details - Summary" and issue resolved. :-)
Thanks,Aaron
I went from Orion Platform 2017.1.1 SP1, VIM 7.1.0, VNQM 4.4.0, DPAIM 11.0.0, NPM 12.1, QoE 2.3, NTA 4.2.2, NCM 7.6, NetPath 1.1.0, CloudMonitoring 1.0.0, SAM 6.4.0
to .... wait, I can no longer scrape that from the bottom of the web window. Must be a graphic element now. Reverting to Soviet mode copy and paste (eyeballs, fingers, and keystrokes):
to Orion Platform 2017.3, VIM 8.0.0, VNQM 4.4.1, DPAIM 11.0.0, NPM 12.2, QoE 2.4, NTA 4.2.3, NCM 7.7, NetPath 1.1.2, CloudMonitoring 1.0.0, SAM 6.4.0
The new downloads are somewhat confusing. I normally go through the Upgrade Advisor to determine the exact products I need to download, and then download the full packages, which I guess is called the offline package now. I saw the note that said
IMPORTANT! The latest releases - NPM 12.2, NCM 7.7, SRM 6.5, UDT 3.3, IPAM 4.5.2, VNQM 4.4.1 and NTA 4.2.3– utilize a new SolarWinds Orion installer that handles product compatibility and upgrades for you.
I tried the Online package, which is marked as recommended. I cannot recommend it after my experience. Originally, I was just going to upgrade IPAM to 4.5.2. The Online package made it clear that it was going to upgrade all upgradable products, and that I would have to wait for each download. So I canceled, and tried the Offline. When I went to apply hotfixes, I realize that it wanted Orion Platform 2017.3 Hot Fix 1, which I think implies NPM12.2.
So I checked the Upgrade advisor, and it does not list IPAM 4.5.2 - only 4.5.1, because 4.5.2 is the new style.
So I proceeded to download the offline for that NPM12.2, ran into various severe errors during install, and concluded that I needed to uninstall, then reinstall various existing, old modules.
But the download page no longer lists the old modules. There's a link for Archived HotFixes, but even it is not complete.
The HotFixes listed for current versions are sometimes odd. For IPAM 4.5.2, it lists IP Address Manager v4.5.1 Hot Fix 1. Do I really need that?
I finally reinstalled all previous versions (because I'm a packrat, and maintain on-disk copies for just this eventuality), and decided to do the online install. I got various Config Wizard errors for almost every product. I pressed on, and finally got a run of the Config-Wizard that came up clean. I don't feel at all comfortable with the result.
Problems I would like to see you address:
Thanks!
During upgrade we do a background inventory to find out if there are some ASA nodes that supports new ASA pollers.
If ASA pollers are discovered for some nodes, we assign new ASA view to their MachineTypes (ViewsByDeviceType) if they are set to “default”.
It looks like there has to be some node with Unknown MachineType (at least before upgrade) that supports ASA pollers, so we have changed a view to “ASA view” for “Unknown”.
If you want us to investigate further, please create a ticket and upload diagnostics.
Thanks,
JP
"because I'm a packrat, and maintain on-disk copies for just this eventuality"
I do this both at work and home.
I always feel bad about doing it (wasting both my time and disk space) and then once in a blue moon it really pays off nicely.
I really wish vendors (everyone who releases code) would always provide a way to get to old releases so I wouldn't have to do this, but I've been bitten just enough to not even trust them if they did
Agreed. It sounds like you've experienced many of the things I did during my upgrade.
More interesting (to me) is the fact that running certain hot fixes followed by the Scaleability Engine updates on the pollers resulted in dual Wizards opening on every APE. Running each simultaneously guaranteed both to fail. Running the first one all the way through to completion, even though the other Wizard popped up part way through, was the path to success. Once the first Wizard was successful, then running the second to the end would work. It's crazy that two would show up at the same time.
Perhaps simpler to fix, but harder to recognize, was that the Wizards would pop up BEHIND the NPM front end page. Definitely a loss of time discovering all the Wizards had started up behind the browser window. I figured I'd messed up somehow, and tried starting them again. Which failed.
By accident I was moving the browser about the RDP window and discovered the Wizard waiting for my input behind ID. Hmph!
Our Upgrade Experience. We have a semi-large environment with 9 additional polling engines and one additional web server. We went from NPM 12.1, UDT 3.2.4, IPAM 4.3.2, NTA 4.2.2, SRM 6.4.0, NCM 7.6 to the latest versions using the new installer that included the latest hotfixes.
I like to wait until the first batch of hotfixes are released before upgrading, which is the reason for our delay.
Ran this upgrade in conjunction with a replacement of the primary polling engine. We did a 2012 R2 server swap for the primary engine. All the other APEs, Fastbit DB server, and AWS have been upgraded prior to 2012 R2 over the last few months. The migration method we followed kept it simple by using the same hostname and IP.
Total outage time was 8.5 Hours.
First 3 hours
We upgraded the NTA fastbit server to the latest version. Next removed the licences from the primary server for our modules and shutdown the services on all the APEs and primary polling engine. We exported the primary server's SSL cert from IIS Manager and copied the legacy reports folder as outlined in the migration guide.
The rest of the first few hours went fairly went well with the primary server hostname and IP swap to a new 2012 R2 node, and the initial install. Loved the "check what you want install" method which automatically included the latest hotfixes. There was one warning on the install where NCM wanted 3.0 GHz processors, instead of the 2.7 GHz ones that are presented to the node. We present 20 cores to the primary engine and have never had a CPU issue so we kept moving forward. One thing that popped up was the NTA install wanted to use a local drive for the initial install, instead of the fastbit server. We did not have the option to select a separate fastbit database server.
This first issue was fixed by downloading and running the seperate NTA 4.2.3 install module on the primary engine and selecting the separate server / database option.
We ran the config wizard 3 x on the primary server, which extended the install time. (1st with initial install, 2nd for NTA installation, and 3rd because the NTA installation caused a website runtime error where we couldn't generate the website). This was the first of two calls to support we had. I opened a case online, then called in and was able to get someone right away. Turned out the fix was to run the config wizard a third time.
Once the primary engine's website was presented we started the upgrade of the 9 x additional polling engines. Eight of these installs went great. The last one turned out to be a problem. We used the new Scalability Installer retrieved from our local Orion server/website. Settings - Polling Engines - "Download Installer Now".
Eight of our polling engines are local to the primary engine, and last one is remote in a separate datacenter. The local APE upgrades all executed without incident. We ran the updates all at the same time and they all completed successfully.
The last remote polling engine kept on timing out, and we opened another support call, first by creating the ticket online then directly reaching out and calling support. Support hold time was about 25 mins before we were able to start working on the issue. We spent the next 4 hours working with support an trying to get the files copied over, circumventing the copy process from the installer. We used a new technique as the one outlined on thwack and in the current KB didn't resolve the issue. Failed to download or run the Scalability Engine installer from the main server - SolarWinds Worldwide, LLC. Help and Su…
Our suspicion was that the KB only dealt with the largest Core MSI, the eventual solution was to follow the KB except that we copied the subinstallers folder out of %temp%\SWOrionSetup, whenever the timeout occurred. Coping the files before exiting the installer allowed us a head-start on the next time we tried the install. We then copied the subinstallers folder back to %temp%\SWOrionSetup and updated the time stamp folder name, then ran the installer again, repeating as necessary until all the installation files were copied.
Working with Brian from support was a great experience. Kudos to him
The last issue that we had was a minor one with the additional web server installation. The initial installation hung on the NCM Integration Module uninstall. A restart of the installer solved that issue.
All-in-all we strayed a half hour out of my planned 8 hour outage window, and I am pleased with the responsiveness of the Website while exploring the new features. I submitted an enhancement request to increase the timeout or allow an offline upgrade for APE.
Thank you for this in depth feedback. We are working on improving the timeout architecture for the remote APEs, so I've flipped that feature request to what we're working on. I hope to hear your feedback the next time you're upgrading to see how we've done to improve that scenario for you.
I think I owe my problem free upgrade to Serena's advice about uninstalling IPAM first. I followed this instruction on my primary Orion server and all additional pollers. I did not see this advice in any documentation and because of my installed versions the Product Update Advisor tool wasn't very helpful this time around. Thank you Serena!
I upgraded from Orion Platform 2016.2.100, WPM 2.2.1, IPAM 4.3.2, NPM 12.0.1, DPA 10.2.0, QoE 2.2.0, VIM 7.0.0, SAM 6.3.0, NetPath 1.0.1 to without any problems at all. I have 2 additional webservers and pollers plus a VMAN appliance and I think I was all done in less than 4 hours. Great experience with the new installer and it definitely pays to read Thwack!
Our upgrade from previous versions went extremely well, the web based online and all in one experience started with a check of the environment. We have an additional poller and a VM with NTA which it detected and told me about the fastbit upgrade which I needed to do after along with the additional poller.
It did the complete upgrade and then run through the configuration wizard. We have
So took all in all about 30-40 minutes (as I wasn't really checking)
Sucked the main poller install down to the additional poller went fine then the config wiz runs.
Upgraded NTA server which went fine
We has warnings about our server:
CPU requirements
DISCOVERED: 15/11/2017 21:33:36
DESCRIPTION: Your environment does not meet the minimum CPU requirements for the following products: Web Performance Monitor 2.2.1. Performance may be slower than expected.
RESOLUTION: For better performance ensure that you have more than 4 CPU cores that are faster than 3000MHz.
but have installed it all - we had issues with a service stopping and stopping after the upgrade but tech support helped which was this, and doing this helped alot!https://support.solarwinds.com/Success_Center/Server_Application_Monitor_(SAM)/Tweaking_performance_of_Windows_Server
Wow this is interesting extra information. I'm wondering now how these settings might help some of my other busy servers that do a lot of network activity!
Few further issues that I'd like to share since our upgrade.
To recap our upgrade included replacing the primary server with a new 2012 R2 node and a clean install. Turns out the installation included some hotfixes but not all. SAM 6.4 HF 2 was not installed and consequently we were unable to gather windows mount volume metrics, to our surprise after the install. The hotfix was installed on the APEs as they were upgraded and not replaced with a new node. Seeing the HF installed on the APEs and not the primary added to the confusion.
SAM 6.4 Hotfix 2 resolved the Windows Mount issue. A few days after that installation we realized we had issues with RabbitMQ not responding and the backup MSMQ overloading and filling up the HDD. This leak also caused intermittent SAM monitors stopping with ephemeral ports being exhausted.
Ephemeral Port Exhaustion - SolarWinds Worldwide, LLC. Help and Support
A complete uninstall and reinstall of RabbitMQ, along with running the configuration wizard on the primary engine with all the APEs shutdown resolved our issues. It was important for all the APEs to be shutdown.
We have been running stable now for a week.
One note I'd like to pass along is I would love to see better release notes that specifically call out which hotfixes are included in the Core roll-up Hotfix bundle.
How unfortunate that parts of, or all of, APE's must be shut down to correctly perform an update or a fix. I'm fighting that same challenge right now.
So how did your own upgrade go?
To be perfectly honest, it is going the wrong direction. The new "manage nodes" interface I shutoff, I revert to the old. OpenGL issues, when I try to open the "new" manage nodes section, that stopped after a patch, but still the interface doesn't let me do any admin work faster. It is all more clicking, more gui navigaation that doesn't let the easy access to custom properties for sorting....total downgrade in GUI.The new "edit views" is awful, hangs up and times out 7/8ths of the time.
Just for some clarity are you saying that the actual product is going in the wrong direction, rather than the upgrade experience itself? Both are interesting points of feedback.
For the new manage entities, there's still a fair amount of work to be done on that page which is why it's not replacing the default views. Until it has feature parity we of course would not replace the default.
The edit views, it sounds like you're running into something unexpected, have you created a support ticket for this?
I can second your observation in regards to the new Manage Entities screen. Once it was corrected so it actually worked, I used it for a while, just to see if there were efficiencies gained by it.
And like you, I shut it off. It is significantly less efficient than the previous solution. It does not display all the nodes on one screen, and it wastes a lot of screen real estate. It'll stay off as long as possible, and I hope the original solution is not removed in future releases in favor of the new Manage Entities screen. Unless, of course, the new Manage Entities screen can show all entities in one screen, and not waste so much space. Which would make it very much like the original solution.
The upgrade went well, the new installer, brilliant. It was nice to only have one file, instead of 4, on each server.
I feel the new Manage Entries page wastes a ton of real estate on large icons, thick rows, and lots of white-space, that annoying square loading box, the new filter part and display make it hard to sort the data. Before I would have polling engine, polling method, or any of the other stuff that makes it easy to sort the list from. I mean, you can add it via filter results, but you could do that already, and then sort again with he columns.I haven't spoken with support about this issue, only a few others. Time is not something that I have a lot of to sit on the phone. I have installed a second instance of SolarWinds this last week for testing, it has the same problems with the Edit views not populating consistently, and the Manage Entries page being a downgrade in functionality. I would like to see some mobile friendly management pages. I would be a lot more supportive of the change if it rendered nice on a mobile. I am often in meetings, or on the other side of campus, being able to pop in and adjust something on the mobile would be great. Currently there is a lot of zooming involved.
I would like to know what functionality it gained, if any. Keep up the great work on the installer.
I'm glad the new installer was back ported from 12.2 to 12.1 as I'm still doing upgrades on some systems from back as far as 11.5. Kinda wild thinking about how far things have come from 10.x now.
I have MANY improvements with the latest code versions, and very few caveats or items left to tweak and tune. I really like the installer that was just updated on 1/16/2018. It fixed every complaint I had in the installation and APE Polling Engine install areas.
Hopefully that'll make some of my upgrades easier then... it's really all about getting off windows 2008R2 and older SQL Server version VM's for everyone now that product doesn't support them anymore.
Upgrades are DEFINITELY easier now. Most of the thinking is done in the background by SW, and they've done a NICE job of it with this week's release/update.
I'm not sure who has access to it yet--my experience seems to have been a beta test, and serena supervised it with me. She indicated that this would soon be published and made available to all, but didn't state when.
It's a keeper!
I was heaving quite a lot of difficulties.
The new installer was not working for me because i have additional polling engines globally deployed and the installer was not able to transfer all the data from my main polling engine to all my additional polling engines.
It took my up to 12 hours to deploy everything and after that NCM was not working because of a bad hotfix.
I'm still not able to run the configuration wizard on non of my systems, it always get stuck in stopping services. I'm really worried that i will no longer be able to get something done.
Also after the upgrade all my polling engines are instable, it seems that more and more of polling engines are running out of diskspace.
So no success for me after more then 1 month i'm struggling to keep everything running.
I hope that the next release will give also the opportunity to have an offline installer otherwise i will not be able to do an upgrade anymore.
I experienced many of those same issues, and a dozen or two Support tickets later they've been resolved. The best solution is the one I recently implemented through the kind graces of serena. I recommend you:
Please try using the new installer that was uploaded this last Monday: file version 1.3.1.493 <-- this contains an updated download algorithm that should speed things up for you. Are you interested in participating in a dev assisted upgrade?
Love this list! Absolutely spot on.
Life might be better (?) if everyone just went out & got the latest download of the Updater/Hotfix . . .
It is for me.
Hey Bartley, what server O/S verions are you running? I am running Windows 2012 server and NPM 12.1. We had extreme difficulties with installers when we were on Windows 2008 R2 server, but as we refreshed to 2012 server we are having less problems. What are the specs on your servers? We have 70 gig C and 40 Gig H on our pollers and all our servers are 8CPU/32 gig of RAM VM's.
We had a similar issue after upgrading to version 12.1 where we had a memory leak. It turns out our problem was related to a bad dashboard. The dashboard was doing a long running query that never finished and spike memory. It has been my experience with the product most CPU and memory issues are tied back to something that we have done environmentally. We have had better experiences after applying all the latest hot fixes.
In the meantime, i think most of issues are solved. The new installer 1.3.1.493 that comes with the latest HF worked fine. datacopy over WAN still takes quite a long for me so the online installer still is causing quite a large downtime, so an offline installer will still be preferable for me.
Also support helped me very good, i think most of my issues are solved now.
All of my servers are running on Windows 2012 R2.
The latest installer and HF solved quite a lot of my issues.
Keep us posted. I also have a Windows 2012 R2 environment. We had a lot of problems with version 11.5 and 12.01. We really stabilized our environment with version 12.1 so we are holding off on upgrading to 12.2. We will probably do version 12.2 in QA soon.
We're upgrading two Orion systems from NPM 11.5.3 all the way up to 12.2 and moving off Windows 2008R2 and SQL 2008 today which means... upgrade in place to 12.1 then move the database over to brand new VM's already running 12.1 then upgrade it to 12.2 with the new installer! Wish me luck friends!!!
All of your help and comments have helped us to plan this all out over the past few months!
You'll have a great upgrade experience, I predict.
The new installers/updaters are much improved over last October.
I wished I'd had SW Support WebExed in to assist my process, but after all was done, and had multiple issues, serena was gracious and kindly provided early access to unpublished installers that cleaned up so many issues across five pollers and all my SW modules (NPM, NCM, NTA, IPAM, UDT, VNQM, HA).
If I could, I'd send myself a message back in time to last September to say "Be patient; wait until February 2018 to do your SW upgrades, and you'll avoid a lot of frustration."
Yay! Even better, all of those unpublished installers are all published and accessible for all to enjoy.
We're in the middle of this upgrade on a couple systems right now. We've gotten hung up on the upgrade of NPM and it's complaining about not being able to communicate with the NTA database. This morning it was working fine on the old versions so no changes to the host machines or anything happened but it won't work now. Installer is saying "Cannot connect to NTA Storage Database" 4.2 when trying to update NTA on the Orion server it cannot can not communicate with the flow storage database. This was all working before here in place. We opened a case and had to leave a message. I don't know the new case number yet. Support didn't send the case number. Support said they'd send an upgrade guide and they didn't do it. We're stuck now with our in production main SolarWinds Orion system down trying to do the upgrade to 12.2. As I'm sure you would understand this is VERY frustrating. We're a large company with thousands of nodes under management.
Thanks for your comments Rick I really appreciate it!!! Hopefully we'll be able to get through this upgrade and have it running before Monday. It hasn't been completely smooth so far :^{
Did you use the NTA standalone installer available in your customer portal for the NTA upgrade? Walk me through the installers that you downloaded and used please
The AZ Solarwinds upgrade is finally complete. My initial outage window estimate of eight hours on Friday was a bit off since it actually took from 0800 Friday until 0400 on Sunday morning (total 44 hours, btw). The problem was that there were both a lot of steps and that these steps took a long time to complete. In fact, though, there was relatively few actual problems encountered, maybe two or three, and all were handled with relative dispatch. Relevant details below:
Modules:
NPM == Network Performance Monitor
NCM == Network Configuration Manager
SAM == Server & Application Monitor
NTA == NetFlow Traffic Analyzer
WPM == Web Performance Monitor
Steps:
This is why we always needed improved installer(s). It is getting better.
Ugh--what an ugly and time-consuming process. Trying to decide when to upgrade is the key, I think. On the one hand you have things working acceptably and people are accustomed to seeing their resources in Solarwinds. On the other hand the product is growing, improving, adding new fixes and features and functionality with every update.
Upgrade too early and you won't benefit from the experiences and modifications brought to the process and the patches and hotfixes by people who discover problems--which may cost you weeks or months of frustrating time working with SW Support. Plus, you have to explain to your peers, your boss, and your customers why things aren't working properly, and when they WILL be working.
Or, wait too long to upgrade and you can end up with versions that can't be upgraded directly to the present release. Which may mean a LOT of intermediary upgrades steps, and additional learning curve and problems you hadn't expected.
Based on my last upgrade from NPM 12.0.1 to 12.2, I'd say that waiting four months from the announcement of the Release Candidate might be a mark I'll try to hit in the future. Waiting through two or more entire versions to upgrade can be challenging when it's time to do the deed.
Previously, I'd experienced so many frustrations that I tended to wait instead of upgrade. But I can see from your endeavour that waiting may be as bad as going through the version-related issues immediately.
My experience is not good. I attempted the upgrade last week, had issues, called in got some assistance but it is still not working. Sent an email to the person I was working with Friday, no response....considering a variety of factors, I was okay with that. Sent another email today and at the end of my day....no response. Not liking this at all. Fortunately for me, I am still running the old version which is a good thing considering I work for a decent size city that cannot afford to be down.
Keep us posted about this. It's important to know what kind of support people are getting. Even my upgrade required a couple calls to SW Support to get through what I did. What really complicated things was having to move the database to new SQL Server and Orion to new Windows servers. 12.2 NPM won't install on Windows 2008R2.
ecklerwr1 wrote:We're upgrading two Orion systems from NPM 11.5.3 all the way up to 12.2 and moving off Windows 2008R2 and SQL 2008 today which means... upgrade in place to 12.1 then move the database over to brand new VM's already running 12.1 then upgrade it to 12.2 with the new installer! Wish me luck friends!!!All of your help and comments have helped us to plan this all out over the past few months!
ecklerwr1 wrote:
Eek - I'm due to run an upgrade from NPM10.7 all the way to 12.2 - They have NPM, NCM, NTA, SAM, IPAM, UDT, WPM & VNQM - This might take a while!
What will your new systems run 2012, 2014, 2016?
I'm using Windows 2012R2 right now.
We were running NPM 11.5.2 on Windows 2008. Installed new servers, Windows 2012; installed 11.5.2 on the new servers (evaluation mode so that we can have it running and tested before pulling the plug on the 2008 servers). Install failed, called tech support. They got us through 11.5.2, then 12.0.1 and then 12.2 but none of the installs were tested until we tried the configuration wizard on 12.2. We were advised to continue the install process; in hindsight, I would ensure that all versions/upgrades were fully functional before moving on to the next version. I am currently on hold waiting for tech support to come on line. I understand having issues (that's why we have tech support) but I had let the person that I had been working with know that I was ready and waiting yesterday morning as soon as I got to work, I did not hear back from them until after I had left yesterday evening that they were going to be out the rest of the week and I should call in.... might be a good reason for that but after not hearing back from them on Friday, I am not real happy. Will keep posting as the install/upgrade continues.
Wow 10.7. Care to share your upgrade experience when you do that upgrade?
There was one case where it wasn't being dismissed but that was fixed in a hotfix. Have you upgraded to the latest hotfixes for the platform?
my installer hangs in "downloading installers for: SolarWInds COre, 2017.3" from the remote sites does anyone here knows on how to resolve this? and do we have offline installers for additional poller?
Just got done with tech support and 12.2 is up and running. Not sure if it's okay to give a shout out to the people that helped but it's working. Learned a couple of things that may benefit others. Previously I said that I would ensure that all versions/upgrades were fully functional-while on the phone with tech support I was told that would probably be a waste of time. As long as the product installs correctly, it is (and was) best to upgrade in the manner that we did; then, if there are problems you fix the final version.
Prior to calling tech support I went into the log files and copied 4 errors that I was getting and pasted them into a file. I shared this with tech support, the one they keyed in was
"Website configuration failed:
• • Connection to SolarWinds Information Service did not succeed.
Last error: The socket connection was aborted. This could be caused by an error processing your message or a receive
timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'."
Tech support advised that we needed to run a file that was a .NET 4.6 fix it tool. I had a hard time following along but was told this was a MD5 issue-this is not a Md5 hashing issue
Due to our network settings, this file could not be downloaded from the server, tech support had me download it and then I copied it over to the server. At that point, I turned control back over to tech support. They extracted and ran it, then repaired the Solarwinds Orion Core Service 2017 (Control Panel). After that we ran the Configuration Wizard again. CF finished successfully this time and 12.2 is running. I was told that I should check all my existing alerts as some may not work the same in 12.2. I logged in to the new web site, I was expecting the initial web page to look very different but it looks about the same. It appears that the main difference is the way we will access adding nodes, alerts, etc.
So depending on your SQL version you could run the upgrade directly on the new box - or at least that is what I'm looking to do.
Totally serena - This is a fairly unique situation as the existing environment has crashed and the new servers deployed are running Svr 2012 R2 with SQL 2014. The customer needs to stand up the existing environment directly onto the new server and then upgrade. So we need to install all the modules to the existing versions (NPM 10.7, NTA 4.0.1, NCM 7.2.2, SAM 6.0.2, IPAM 4.0, UDT 3.0.2, VNQM 4.1, WPM 2.0.1). After a little bit of research it turns out SQL2014 won't support that but SQL2012 SP1 will. So I've got the NCM and NPM DB's restored to the SQL2012 Server and then we will install all the products on existing versions onto the new 2012R2 Server and once that's all working go through the 20+ steps to upgrade to latest versions.
What file version of the installer are you running?
If it's the latest for instance 1.3.2, then you can navigate to C:\ProgramData\SolarWinds\Installers on your main poller. Copy the CoreInstaller.MSI (the file that the installer is hanging on) and you can copy that over to the installer folder on the additional polling engine. The core file is usually the largest, so takes a while. However, it will eventually complete. by moving the file over manually, you're just going to make it a manual process.
C:\ProgramData\SolarWinds\Installers on your main poller. Copy the CoreInstaller.MSI (the file that the installer is hanging on) and you can copy that over to the installer folder on the additional polling engine. The core file is usually the largest, so takes a while. However, it will eventually complete. by moving the file over manually, you're just going to make it a manual process.
Glad you made it through it... hopefully going forward with newer installers we'll be in better shape now.
Hi orioncrack,
We have done 2 times. First time it was successful and we went from NPM 11.0.1 to 12.2 along with NTA, NCM. No additional poller. Here, I found the Orion installer as very useful and it was smooth.
Last week, we did that from 11.5.3 with other modules and additional poller. We could upgrade till 12.0.1 only. All other modules up to next version and NTA to 2 vrsions (to 4.2.1). If you have additional poller it would take lot of time (depending on your DB size) and lot of checks are there with dependencies you need to upgrade old versions first, to start the Orion installer from 12.1. So, we are doing the remaining upgrade next weekend to be with the latest versions.
More notes on my upgrade experience. To recap, attempted installing 11.5.2 in eval mode on new servers-failed. Tech support got me through the major portion of the upgrade. Once we got to 12.2 and ran the Configuration Wizard, we got a failure. Contacted Tech Support again, different person and they got me the rest of the way.Here are the final notes that were sent to me that outlines the basics of what happened on the last call:
- Checked SW certificate.- Using md5- Use Fixit tool.- Now using SHA1.- Run repair core.- Orion Web is now up.I had to correct 3 nodes that didn't transfer correctly. I had been informed that some alerts may have issues also, this turned out to be true. I have been able to find those under the Notifications tab (the bell beside your username) and on the Alert Migration Log I can see all the alerts that transferred, the ones that failed and are unusable, there are other alerts that I am informed that need to be looked at (message: Object/properties used in trigger/reset conditions of this alert are not supported in new web-based alerting.) . I had heard that the web console may look very different, this was not the case for me. All in all, I am satisfied with the new upgrade-I have a lot of work to do with the alerts that either didn't migrate or have trigger/reset condition issues. Really like the Unmanage method as well. Will update if I see anything else of significance.
You may as well move to SQL 2016 now, since earlier versions won't be supported by NTA. Save the hassle and be prepared by putting in the new version, and your upgrades won't run into that snafu.
Hi CRZYFST,
I am very interested to know how you upgraded your servers and sql from 2008 to 2016 because I am in the process of doing an in-place upgrade for both from 2008 to 2016 before upgrading our 5 SolarWinds products to the latest version, but I'm very nervous.
Did you do a direct upgrade from Win server 2008 to 2016 and SQL server from 2008 to 2016 successfully? Migration to new would have been easier but mine is same machine (in-place)
Your response is highly appreciated.
Glory
We are migrating to NPM v12.2, NCM, SRM, WPM, etc. from NPM 11.5.3. on WIN 2008 and WIN 2012 servers to WIN 2016 and SQL 2016.
What is great about the server migration, is that we can build out our entire new environment whilst old solarwinds chugs away. Once the database is transferred over from the old system and built up, and all reports, etc. verified we simply, on the night of the change, transfer back to our real licenses, switch urls, and go live with little to no risk and little to no downtime.
The tricky part is you can't update to NPM12.2 on windows 2008R2 so you can only upgrade to NPM12.1, then move the database over to new NPM12.1 server on windows 2016. After the move you can then update to NPM12.2 and associated modules with the new installer. We're doing a second iteration of my above upgrade today. Hopefully it'll be a little smoother since doing the first one a couple weeks ago.
So I have two complaints so far:
1) The Product Upgrade Advisor couldn't give me a clear upgrade path - it came back stating too complex! When I logged a support ticket I didn't really get the answer I was looking for.
2) I used the 12.2 Installer on the box to help guide me on my upgrade path, which worked splendidly and produced a full upgrade path step by step, however when I got to NPM12 it told me I was unable to progress due to 2 or 3 modules not being up to date but the original advisor hadn't listed them. Once I installed those modules and re-ran the 12.2 Installer again it gave me a slightly different install path. Now I am not sure if I can trust the built-in advisor of the all in one installer.
This is all based on lab testing, the production upgrade is not until next week so hopefully, I have time to tweak and test it.
We successfully updated the second Orion server to NPM12.2 and associated modules. One thing I forgot from the first time and had happen again is when you change the server name for your primary poller... support has a script that you run which goes through the entire database and basically updates any references to old server name to the new server name. If you (like me) have to move the SQL server database and it's to a host with a new name you will need this script.
Wait, so not even SQL 2014 will be supported?
When I was upgrading I found RC versions and also strange we were facing lot of issue but sw tech support team hasn't identified that beta had been installed in my network. Do you know to check this versions issue in Orion polling engine like it's RC or Beta Or SP or GA?
I believe that the RC issue is resolved, are you still seeing this behavior?
@ rschroeder were you able to upgrade from 12.01 to 12.2, skipping 12.1? I was trying to upgrade 12.01 to 12.1 in my lab environment and it corrupted my installation. I'm re-installing 12.01 and then going to try going directly to 12.2 this time.
