48 Replies Latest reply on Jul 22, 2015 2:37 PM by Manilyn Ramos

    Upgraded to new release of NPM 11.5.1 last night

    fitzy141

      So after upgrading the primary , secondary poller and additional web server on recommendation from support to address the multiple of bugs I am having - I know see this  error from time to time   ( Server error: Error: A query to the SolarWinds Information Service failed. ) and when watching the Orion services manager solar winds information service V3 keeps going up and down ...

       

      SO much for painless upgrade again

        • Re: Upgraded to new release of NPM 11.5.1 last night
          deverts

          fitzy141,

           

          I'm assuming you upgraded from 11.5 to 11.5.1? Like most, I've been wanting to upgrade to 11.5 for all the new cool features, but I'm waiting because of all the issues I've seen posted. I'm curious if the upgrade path from 11.1 to 11.5.1 is any better.

           

          Also, did you try re-running the configuration wizard to repair the services only? I've seen several people mention they had to run the wizard twice to fix certain things.


          D

          • Re: Upgraded to new release of NPM 11.5.1 last night
            prawij

            fitzy141, did you have trouble with the Information Service v3 right from when you upgraded to NPM 11.5, or did it start occurring after you installed Orion Hotfix4 or NPM HF1 by any chance?

             

            Once we deployed Orion Hotfix4 and NPM HF1, the only way we could get things back up and running on our pollers was to restore the following directories from a backup taken prior to applying those hotfixes (strangely enough HF3 never broke anything for us):

             

            C:\ProgramData\SolarWinds

            C:\Program Files (x86)\Common Files\SolarWinds

            C:\Program Files (x86)\SolarWinds

             

            Without that radical step the Info Service 3 would just keep going down and up by itself. Sounds like 11.5.1 may have the same issue......

            • Re: Upgraded to new release of NPM 11.5.1 last night
              ecklerwr1

              Ugh!  please keep us posted on the progress with this... it's very important to the rest of us as I'm sure you can imagine!  o.O

                • Re: Upgraded to new release of NPM 11.5.1 last night
                  fitzy141

                  I ran the config wizard again on just the services last night and have ben watching it the service manager on and off during the day so far today . will let you know

                    • Re: Upgraded to new release of NPM 11.5.1 last night
                      nickepa

                      fitzy, Are you service(s) stable?  I ran the config wizard a well and am still getting the same errors (I believe everyone else is getting).  I do have a SR open as well that i'm waiting to hear back on next steps.

                       

                      Faulting application name: SolarWinds.InformationService.ServiceV3.exe, version: 2015.1.0.227, time stamp: 0x55357860

                      Faulting module name: clr.dll, version: 4.0.30319.34209, time stamp: 0x5348a1ef

                      Exception code: 0xc00000fd

                      Fault offset: 0x000000000001da5e

                      Faulting process id: 0x2b14

                      Faulting application start time: 0x01d082a942f4c710

                      Faulting application path: C:\Program Files (x86)\Common Files\SolarWinds\InformationService\V3\SolarWinds.InformationService.ServiceV3.exe

                      Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll

                      Report Id: b8f2ca3a-eea2-11e4-80e3-0050569c1c79

                      Faulting package full name:

                      Faulting package-relative application ID:

                       

                       

                       

                      For fun I ran the Solarwinds Active Diagnostics and it reported the following (see below).  However, I doubt these messages are related to one another.

                       

                      SolarWinds installation files are different.

                       

                       

                      ============================================================

                      SolarWinds Orion Network Performance Monitor v11.5.1

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Collector v2.11.207

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Orion Improvement Program v2.0

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Job Engine v1.6.0

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Orion Core Services 2015.1.1

                      ============================================================

                      Package File: \SolarWinds\InnovaDSXP2.ocx

                      Disk File: C:\Program Files (x86)\Common Files\SolarWinds\InnovaDSXP2.ocx

                      File sizes are different. Package: 1379043 Disk: 1383192

                      File hashes are different. Package: a1360473c0ea0925ebf3fefcb57f10e82d198415 Disk: cb18ae063c6f3cceeb05ce1efd167aacc85ed0c5

                      File versions are different. Package: 2.0.0080 Disk: 2.1.0144

                       

                       

                      Package File: \sqlunirl.dll

                      File sqlunirl.dll not found on system.

                       

                       

                      Package File: \SolarWinds\Orion\ModulePatches\UDT-3.0.2-patch.zip

                      Disk File: D:\Program Files (x86)\SolarWinds\Orion\ModulePatches\UDT-3.0.2-patch.zip

                      File sizes are different. Package: 4098 Disk: 1410

                      File hashes are different. Package: 6456a50e22bfe42c464608cd28b89becaaa4ebfd Disk: 583ac3c66a891031d278d67d53ee8eac3e31b9c3

                       

                       

                      Package File: \Win\System\richtx32.ocx

                      File richtx32.ocx not found on system.

                       

                       

                      ============================================================

                      SolarWinds Information Service v2015.1.227

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Orion Network Atlas v1.13.448

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Active Diagnostics 1.1.0.230

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Job Engine v2.9.0

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Server & Application Monitor v6.2

                      ============================================================

                      Package File: \APM\SmartThreadPool.dll

                      Disk File: D:\Program Files (x86)\SolarWinds\Orion\APM\SmartThreadPool.dll

                      File sizes are different. Package: 73728 Disk: 60928

                      File hashes are different. Package: bbc55475ebbf4826ea5af8952d222f2969293f1f Disk: c4d266a29fcd78e23b7f008970e6f8958cc670ba

                       

                       

                      ============================================================

                      SolarWinds Agent 1.1.727.0

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds Integrated Virtual Infrastructure Monitor v2.0.1

                      ============================================================

                      No file differences found.

                       

                       

                      ============================================================

                      SolarWinds MIBs v1.0.5

                      ============================================================

                      One or more errors occurred.

                      Could not find a part of the path 'C:\Users\user\AppData\Local\Temp\47e7b6c6'.

                       

                       

                      ============================================================

                      Files Installed by Multiple Products

                      ============================================================

                      Disk File: D:\Program Files (x86)\SolarWinds\Orion\ModulePatches\UDT-3.0.2-patch.zip

                        Product: SolarWinds Orion Network Performance Monitor v11.5.1

                        Product: SolarWinds Orion Core Services 2015.1.1

                  • Re: Upgraded to new release of NPM 11.5.1 last night
                    silverbacksays

                    awesomesauce.jpg

                    Hopefully it's something simple and support can sort it out for you. I'm mostly running 11.0.1 on the instances I support, and like many of us have been holding off 'till 11.5.1.

                     

                    Has anyone had any problems going from 11.0.1 > 11.5.1 as yet?

                    • Re: Upgraded to new release of NPM 11.5.1 last night
                      edmaendel

                      My 11.5 to 11.5.1 upgrade went very smoothly yesterday.

                      I also went to the NCM 7.4 RC at the same time.

                      Now running: IPAM 4.3, NCM 7.4, NPM 11.5.1

                      • Re: Upgraded to new release of NPM 11.5.1 last night
                        purdue98

                        I was experiencing this same issue but my issue was resolved.  Solarwinds provided a fix.  The engineer said it was a workaround while they finish the more permanent fix.  After the workaround, I haven't seen any errors about querying the Information Service and my Information Service has been up for about an hour.

                        • Re: Upgraded to new release of NPM 11.5.1 last night
                          clubjuggle

                          There is a note in NPM 11.5 rollback process indicating that Solarwinds support now has a fix for this issue.

                          • Re: Upgraded to new release of NPM 11.5.1 last night
                            oiram

                            You may find below HF handy for 11.5.1

                             

                            http://downloads.solarwinds.com/solarwinds/Release/HotFix/OrionPlatform-v2015.1.1-HotFix1.zip

                             

                             

                            SolarWinds Orion Platform 2015.1.1 Hotfix 1

                            ------------------------------------------------------------

                             

                            This Hotfix addresses these issues:

                            1. SWIS V3 Crashing since upgrade to NPM 11.5.1

                             

                            Apply this Hotfix if you run any of the following SolarWinds products:

                            - NPM 11.5.1

                            - UDT 3.2.2

                             

                            Notes:

                            - This Hotfix requires SolarWinds Orion Core 2015.1.1.

                            - This Hotfix must be installed on the main poller.

                            - This Hotfix must be installed on any additional pollers/webservers.

                            - This Hotfix contains the InformationService.msi file with InformationService version 2015.1.236.

                            - This Hotfix addresses issues for all SolarWinds products located on the same server. As a result this Core platform Hotfix only needs to be applied once per server.

                              • Re: Upgraded to new release of NPM 11.5.1 last night
                                ctmidnight

                                Sadly I have all the latest updates except the NCM 7,.4 RC installed, and things are pretty bad since our largest system was upgraded.

                                The upgrade of our external web servers all bombed at the end of the config with 403 errors, and repair of any of the installed components like this one for IPAM produce the same error.

                                At present we cannot use the product at all, since the MSMQ stack maxes out on athe primary (and presumably all the APEs as well) because of communication issues through the API for virtually anything.

                                The database is up and responding fine on this affected instance, but it appears that we may have corrupted data in the alert and node tables since the upgrade in addition to the Information Service fun.

                                No response to our growing stack of support tickets either, and our SVP is preparing her "we're gonna rip this thing out" email to the rep.

                                 

                                Website configuration failed:

                                  •  Web Request for /Orion/Login.aspx failed -  The remote server returned an error: (403) Forbidden.

                                  •  Connection to SolarWinds® Information Service did not succeed.

                                Last error: There was no endpoint listening at net.tcp://pc1wprodswc01:17777/SolarWinds/InformationService/v3/Orion/certificate that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

                                • Re: Upgraded to new release of NPM 11.5.1 last night
                                  bscherer

                                  This is what support sent me this morning, fixed my issues. 

                                  • Re: Upgraded to new release of NPM 11.5.1 last night
                                    fitzy141

                                    yea so far seem ok after the another hotfix that support sent me

                                  • Re: Upgraded to new release of NPM 11.5.1 last night
                                    RichardLetts

                                    Results of our upgrade:

                                    a) alerts table update broke some reports.. not that much of a problem

                                    b) one additional poller appears broken-- with SW Dev to figure out what happened there

                                    c) misc. config files stomped over by the installer -- separate cases opened for each of these.

                                    d) cannot add any new customize/add resources to a page ("Object reference not set to an instance of an object." errors)

                                     

                                    so, I give this upgrade a B+

                                      • Re: Upgraded to new release of NPM 11.5.1 last night
                                        epenney

                                        Richard,

                                        we are considering upgrading from npm 11.0.1 -> 11.5.1 next week. Now that you've been on it for a few weeks, are you glad you upgraded, despite the issues you experienced, or do you wish you waited? appreciate you thoughts.

                                        Ellen.

                                          • Re: Upgraded to new release of NPM 11.5.1 last night
                                            RichardLetts

                                            Our environment is probably one or two orders of magnitude larger than most customers

                                            a) has been fixed by me re-writing reports; the new SDK documentation helped with that

                                             

                                            b) I deleted everything solarwinds-related on the servers, moved the directories to one side and did as fresh an install as I could and the problem was resolved.

                                            really, FOE is a pain in the ass to deal with, and I understand what it's doing as we use neverfail elsewhere but I'm not convinced that it's the right failover technology.

                                            [really allowing more dynamic moving of polling from one server to another would be a better solution, but APE cost way more than FOE...]

                                             

                                            c) This is annoying, and caused us to lose all of our historical AP data. I can't restore the database because it has the wrong schema. it also broke the topology calculator and business layer, but thos were easily fixed.

                                            d) deleted the inetpub directory and re-ran the install wizard to recreate. fixed the problem.

                                             

                                            I do like

                                            a) the new report writer, especially the 'this will general 1000 alerts if you enable this alert now' widget at the end that tells me I've messed up a condition before I light up everyone in the NOC.

                                            b) the ability to define new pollers to cope with odd machine types. I'd like this to be easier to use, but I now have all our werid new stuff polling and being 'correct' rather than as generic net-snmp devices.

                                             

                                            In general I'm okay that we upgraded, like I said, I'd give it a B+. I needed to do it so i can get the UDT upgrade scheduled

                                            I would suggest you go to 11.5.2 as that includes some significant patches.

                                              • Re: Upgraded to new release of NPM 11.5.1 last night
                                                jbiggley

                                                Curious as to what "one or two orders of magnitude larger" means

                                                 

                                                We have 18 pollers across 3 data centers, 11000 nodes, 75000 elements, almost 100,000 application monitors.  Running NPM, SAM, VMAN, NTA (kind of) and just purchased STM/SRM.

                                                 

                                                Is it that kind of big? I ask as the impact of a botched upgrade on a single poller environment stinks.  When you magnify that across 18 pollers it makes life miserable.

                                                  • Re: Upgraded to new release of NPM 11.5.1 last night
                                                    RichardLetts

                                                    You're also not most customers... The University of Washington is much larger than people realize, and the network we mange is even larger.

                                                    I only look after the network infrastructure so: we're five pollers (in FOE, making ten upgrades each time we bump a version) across 3 data centers, 6700 nodes; 38,000 elements,

                                                    the nodes includes 20 wireless controllers that support over 9000 wireless access points, on any given day we have about 180,000 wireless devices roaming around the network.

                                                    Here is just a fraction of our network:

                                                    OneMap.png

                                                     

                                                    so, about half your size.

                                              • Re: Upgraded to new release of NPM 11.5.1 last night
                                                tigger2

                                                "cannot add any new customize/add resources to a page ("Object reference not set to an instance of an object." error"

                                                 

                                                I had this bug in earlier versions of NPM.  If you want to give it a shot, he fix (hopefully for you too) may be here: Re: Object reference not set to an instance of an object.

                                              • Re: Upgraded to new release of NPM 11.5.1 last night
                                                txyang

                                                this HF applies if you are running either NPM 11.5.1 OR UDT 3.2.2, yes? We don't have to have both installed for the HF to apply? Thanks

                                                • Re: Upgraded to new release of NPM 11.5.1 last night
                                                  ecklerwr1

                                                  Btw my comment before about EOC was a crack on EOC :^}

                                                   

                                                  The other EOC users know what I'm talking about o.O!

                                                  • Re: Upgraded to new release of NPM 11.5.1 last night
                                                    ctmidnight

                                                    This hotfix should resolve the info service crashing issue, but it is a memory hog and performance is not so good overall.

                                                    The resulting "upgraded" Information Service V3 should appear in Control Panel after the change as: