19 Replies Latest reply on Nov 18, 2014 8:54 AM by rob.hock

    When do you create a service release for Orion?

    Deltona

      Hi,

       

      I am curious to know why there hasn't been made an service release for NPM v10.7? As of today, there are 9 hotfixes for NPM v10.7. Installing these hotfixes required alot of planning and time to execute, especially when these need to be deployed on multiple polling engines and additional web servers.

      Such an embarkment takes more that half a day to complete! Support suggests to just update the system to NPM v11 when it comes out. Well today it has been released but this is a whole new version that needs to be tested first, pass QA then PoC'ed for a few weeks. It will take a month before we can get clearance for this so the "just do an upgrade" is not possible.

      It seems like hotfix and service releases have become a joke and aren't being taken seriously anymore. Could someone please provide details on the release procedure at solarwinds for hotfixes and service releases. When is a service release created? Is it when there are more than X hotfixes or what?

        • Re: When do you create a service pack for Orion?
          cobrien

          Hot fixes and service releases are certainly important to us.  We released 9 hotfixes for NPM 10.7, as you mentioned.  I'm not sure there is any official process to hot fixes or service releases.  We weigh situations as they come and try to make the best decision for our customers.  It sounds like in this case it would have been better for you to have a service release for 10.7.  I'd like to understand that better.  In your environment, it sounds like the testing requirements for deploying a new major release are different vs a hot fix or a service release.  Can you elaborate on that a bit?  That would help me understand your situation better.

            • Re: When do you create a service pack for Orion?
              Deltona

              Hi cobrien,

               

              We only run software in production if it runs without any issues. We are deeply dependant on Orion and we therefore expect it to perform without trouble 24/7/365. This then requires in house testing to confirm that there are no issues which would cause downtime, wrong data or performance issues. We have 0 tolerance for software that doesn't deliver.

              If there is a known issue and that issue can be addressed with a hotfix then that hotfix would be tested and confirmed to solve the issue as well. Applying a hotfix to a test setup for testing is rather straight forward and in best case scenario would take very little time to carry out. Applying a hotfix to a production system is a whole different story. Not only are we required to create change requests and change documentation but we are also required to test the change for at least a week before going live. 9 hotfixes would then require 9 change requests, 9 change docs, 9 testing procedures, 9 weeks of testing and then 2 more weeks to implement on a production system. You probably think that this is overkill and feel sorry for us, we do too but these are the guidelines we need to follow...

               

              When it comes to deploying a version release, the procedure is similar to hotfix testing, only it takes much longer because bugs and issues are expected to be present in such a release. Only when a service pack is made available do we ever consider starting a test. Example; with NPM 11, we'll only consider testing it for production use once 11.1 or 11.x is released.

               

              Again, we are not the only ones that have expectations to the software that we use. If it doesn't deliver then we won't be using it. Hope that explains the situation.

            • Re: When do you create a service pack for Orion?
              rob.hock

              Just to add to cobrien's response, HFs are very narrow in scope and impact, and in general shouldn't be applied unless you are experiencing the issue that the HF addresses. The critical mass for service release would essentially be a sufficient number of high-impact issues that apply to a broad section of the user-base. This also weighs against how close the next major-release would be that would consume those fixes as well. To your point about requiring time to QA an upgrade, it would be frustrating I'm sure to start QA on a 10.7.5 release just to have the 11.0 release 90 days later.

                • Re: When do you create a service pack for Orion?
                  Deltona

                  Hi Rob,

                   

                  Call it an extreme coincidence but all the issues that the hotfixes address, bar one, exist on our setup. As far as the 10.7 hotfixes provided so far I can see that all of them are critical to a functional (one that is used by people) deployment. Let me prove it:

                   

                  Network Performance Monitor v10.7.0 - Hot Fix 1 for Daylight Savings Time problem

                  This HotFix addresses an issue where polling stops before the shift to the daylight saving time due to the effects of the time change on scheduled polling jobs.

                   

                  Network Performance Monitor v10.7.0 - Hot Fix 2

                  This hotfix addresses an issue when adding a node via web console add volumes with "Volume Type" which depends on browser locale. "Volume Type" should always be in Orion's primary locale.

                  Not having "Volume Type" in the primary locale can for example break alerts filtering.

                   

                  Network Performance Monitor - Hot Fix 3 for OpenSSL issue

                  This HotFix addresses the following security issues that occur in OpenSSL: CVE-2014-0224, CVE-2014-0221, CVE-2014-0195, CVE-2014-3470, CVE-2014-0160, CVE-2014-0076, CVE-2013-4353, and CVE-2013-6450.

                   

                  Network Performance Monitor v10.7.0 - HotFix 4

                  This HotFix provides a patch for the DB maintenance overdue issue that is caused by Interface Baseline Calculation. This HotFix applies to SolarWinds Orion NPM 10.7.

                   

                  Network Performance Monitor v10.7.0 - HotFix 5

                  This hotfix addresses the issue with parsing the VRF Route Distinguisher on some Juniper devices.

                   

                  Network Performance Monitor v10.7.0 - HotFix 6

                  This HotFix addresses following three error messages visible in dataprocessor.log.
                  1. System.InvalidCastException: Specified cast is not valid.

                  2. NPM.Node.MulticastRouting.Snmp failed. System.OverflowException: Arithmetic operation resulted in an overflow.

                  3. System.Data.SqlClient.SqlException (0x80131904): Cannot insert the value NULL into column 'MulticastGroupNodeID', table 'SolarWindsOrion.dbo.NPM_Multicast_RendezvousPoints'; column does not allow nulls. INSERT fails.

                   

                  Network Performance Monitor v10.7.0 - Hot Fix 8

                  This HotFix improves using Active Directory Groups website login. It introduces configurability of used authentication and Group Membership evaluation methods.

                  Symptoms: The user is not able to use an Active Directory Group account to log in to the Orion Web Console.

                   

                  Network Performance Monitor v10.7.0 - Hot Fix 9

                  This HotFix addresses an issue where some nodes are not polled in NPM 10.7 until you restart the Collector service.

                   

                   

                  We would much rather start QA on a service release than on a version release because as with all new version releases, they WILL include bugs! That is mainly because new features are implemented in that new version. This is why the stable release concept is important with software that users are heavily dependant on. We always favor stability above and instead of new features.

                  • Re: When do you create a service pack for Orion?
                    sja

                    HI Rob

                     

                    I will sure support Deltona on that.

                    Why 9 hot fix and not 15 ? ;-) if there no guideline.

                    Sure like to have some officel guideline

                    1.up to X hot fix per GR.

                    2.up to X days/weeks from  last hot fix to SR

                    3.up to X SR to new GR

                    4.major bugs should not move from GR to GR.

                     

                    That will sure help the admin that running large environment  to take the call.

                    • Re: When do you create a service release for Orion?
                      Deltona

                      Hi Rob,

                       

                      How likely is it that a 10.7.1 release will make it to the portal?

                        • Re: When do you create a service release for Orion?
                          rob.hock

                          With the release of 11.0, we would likely not have a SR for 10.7, as we typically roll previous fixes into the new major release.

                            • Re: When do you create a service release for Orion?
                              sja

                              Hi Rob

                               

                              What Dev /you opinion on about schedule SR?

                              What Dev /you opinion on about some "time frame" guides  from GR ->SR?

                               

                              /SJA

                                • Re: When do you create a service release for Orion?
                                  rob.hock

                                  With the dev / QA overhead associated with a release, it is more a matter of having a technical driver for a SR than a time-bound event. We can certainly discuss with our dev management team to see if we can proceduralize for the future.

                                    • Re: When do you create a service release for Orion?
                                      Deltona

                                      Rob,

                                       

                                      This is an extremely important feature of the product that should have maximum attention.

                                      Dominoing patches and hotfixes forward into new version releases might be a good exercise to keep everyone "on the boat" but this makes the product certifiably unstable - forever!

                                      We've had stable builds with regular service releases that updated the version code in the past. The stable releases were crucial in the trust building of the product and have been key to keeping customers.

                                      What happened?

                                       

                                      I can't believe how difficult it has become compared to the structure of old. I have no idea what hotfixes have been applied as Add/Remove programs doesn't get updated with patch version codes. Speaking of patches, the hitfixes listed above are in essence patches.

                                       

                                      Please push for a stable release, please!

                                        • Re: When do you create a service release for Orion?
                                          rob.hock

                                          Please don't take this to mean we would not have service releases going forward, and we are also looking at alternate delivery methods for HF for the future. To your point, it would be advantageous to make it easier to understand what HF are applied to the environment. 

                                            • Re: When do you create a service release for Orion?
                                              Deltona

                                              All I know is that this setup is going to be running crippled and can't be used properly for the next couple of months because the patches/hotfixes/bugfixes we require are only present in the next version release (required patch is not one of the 9 released hotfixes).

                                              We will inevitably find a bug post Q1/2 2015 after finally having upgraded to v11 and that bug would turn out to be addressed in the next version release, v12 maybe. As for the 1 version + 2 service release cycle model, all we are guaranteed to get annually is 1 version release that is guaranteed to have bugs.

                                              Sure it adds up but not to what you'd expect.

                                • Re: When do you create a service pack for Orion?
                                  cgregors

                                  I also support the concept of Service Packs.  Using our friend Microsoft as an example, they roll a large quantity of patches (hot fixes) into a Service Pack that has been vendor validated to work together and has a pretested installation sequence builtin.

                                   

                                  Please consider the idea of Service Packs using one of the following criteria:

                                  • We've not produced a hot fix for 3 months. Lets roll all the fixes together into a SP
                                  • By policy, we release service packs every 6 months.

                                   

                                  Either criteria would make Orion administrators lives easier when patching and/or justifying patching to management.

                                   

                                  Chris

                                  • Re: When do you create a service release for Orion?
                                    sja

                                    Hi Rob

                                     

                                    Now NPM 11.0.1 is out very fast or too fast ?

                                    I can see  support to SQL 2012 sp2

                                    but no bug fix ...

                                    I still don't get the logic ....

                                    when a update like that could be a hotfix and when it's SR or GR...?

                                     

                                    I know you can say no logic..

                                     

                                    :-)

                                     

                                     

                                    /SJA

                                    • Re: When do you create a service release for Orion?
                                      cahunt

                                      There should be serious consideration for Deltona 's concerns. As I too am part of an environment where patching is acceptable and understood.

                                      BUT

                                      Moving to a new Version requires paperwork,scope, testing, planning, a NEW Risk Assessment, implementation and further checking to verify everything is running properly.

                                       

                                      It is much easier for us to update the current version rather than dealing with the paperwork of the so called 'Upgrade'.

                                       

                                      MS still updates/patches SOME products even after the new version has been released ... this is merely reference of concept as we certainly do not want to use MS testing and/or release structure as a model for SW Products.

                                      • Re: When do you create a service release for Orion?
                                        sja

                                        So what's up solarwinds ?

                                        NPM 11.1 has 7 HF and know bugs from 10.7 RC

                                         

                                        Sure like to see 11.2 SR that fix some old issues....