9 Replies Latest reply on Jul 3, 2009 2:34 AM by pserwe

    Observation: General lack of testing environments

    Miron

      Hi,

       

      As with many others on the forum I jump at the opportunity to upgrade to the latest Solarwinds installation and as long as I have backed up the necessary elements I am happy to continue on our production systems.

      On this occassion I luckly I have not taken the plunge not because I expected failures but because I want to get the SQL database off the servers as they are at the moment as its killing the performance - before going ahead

      What has occurred to me during reading all the issues and problems our peers are having with the new upgrade is that it seems that in general network monitoring does not seem to be high on the list for being afforded a testing or pre-deployment environment as you would expect from other business critical applications and services. (My organisation included)

      Granted there are some obvious issues with having a replica pre-deployment enviroment such as cost for mirroring the system.

      I think that possibly out of the perceived chaos of the recent and well intentioned upgrade it will provide the opportunity to add weight to a business case for ensuring that the resources are allocated to provide an appropriate testing enviroment.

      :-)

      Miron

        • Re: Observation: General lack of testing environments

          +1

          It would be nice to be able to test, and not by having to install the demo each time a patch is released.  Speaking from experience, I have had a number of issues with the upgrade to 9.5.

            • Re: Observation: General lack of testing environments
              neilmborilla

              +100%

               

              It looks like the end-user or customer will do the test for them....

              This statement might be harsh to them but it's a reality....

               

              In my case, I haven't experienced smooth upgrade.

                • Re: Observation: General lack of testing environments
                  denny.lecompte

                  It looks like the end-user or customer will do the test for them....

                  Well, yes and no.  We do lots of testing internally.  But we have many many thousands of customers with a wide variety of environments, and our experience has been that no matter how much we test internally, we find new bugs when customers get the release. 

                  We did a beta and a release candidate and found some bugs, but we not as many as we wanted to find. 

                  So I'll note that most people have had good upgrades, but that many of the people with bad upgrades write about them on thwack, which is natural and great and why thwack exists, but it also leads to a slightly erroneous perception that the release is bumpier than it really is.

                  Nevertheless, the release was not as smooth as we wanted it to be.  We are planning some changes to our process that will involve more people putting into production before we release it more generally.  The goal is let people select themselves into the program.  Those who want the code early and who are willing to take some bumps can do so.  By the time we make it more widely available, it should be smoother.

                    • Re: Observation: General lack of testing environments
                      neilmborilla


                      So I'll note that most people have had good upgrades, but that many of the people with bad upgrades write about them on thwack, which is natural and great and why thwack exists, but it also leads to a slightly erroneous perception that the release is bumpier than it really is.

                       



                      Good thing about it is that SW staff is very responsive in dealing with customer's cases. From lowest rank to director =)

                      Its some sort of a relief when a customer's complaint are being addressed and not being ignored

                        • Re: Observation: General lack of testing environments
                          denny.lecompte

                          Miron,

                          I didn't take your post as a criticism.  I just used it as an opportunity to get the message out that we're sensitive to what's going on and are taking measures to improve our release process.

                          Your point is an excellent one.  A test lab is an invaluable resource.  For a product like Orion, it's worth the time to copy the DB upgrade the copy, and check for issues.  It saves any surprises from hitting your production environment.  Although it's getting better, not everyone has a test lab yet.

                            • Re: Observation: General lack of testing environments
                              kulastone

                              I agree with the opinions being aired here on this topic. NPM has become a widely used application that no one can live without. We personally have a test environment that we test almost all upgrades or new implementations before pushing it into production.


                              It would be great if Solarwinds allowed for temp licenses for a week or something so we can carry out testing on our parts. I believe every users setup is different and a lot more can come out of this.

                              • Re: Observation: General lack of testing environments
                                pserwe


                                Miron,

                                I didn't take your post as a criticism.  I just used it as an opportunity to get the message out that we're sensitive to what's going on and are taking measures to improve our release process.

                                Your point is an excellent one.  A test lab is an invaluable resource.  For a product like Orion, it's worth the time to copy the DB upgrade the copy, and check for issues.  It saves any surprises from hitting your production environment.  Although it's getting better, not everyone has a test lab yet.

                                 



                                Denny, Miron, anyone else:

                                While the luxury of a test lab would be awesome to have, there are a couple of issues with it that are pretty difficult to overcome.

                                1)  $10-15k worth of hardware.  To properly mirror production, there needs to be the same infrastructure in place.

                                2)  Configuration changes on every single device managed.  I don't know how everyone else does it, but all of my devices need to have my Orion server's IP set in them as both the poller and the trap receiver.  For my customer CPE devices, this is a configuration change, and while there's an appropriate protocol for handling that, it involves pre-notification, window being notification, window end notification.  There is a lot of labor involved in that process.  The cost would easily be twice that of an annual maintenance cost.

                                3)  The idea of a slower release cycle with a more gradual move from ED (early deployment) to GA (general availability) is a great one.  8.5 to 9.x was fast, but it worked pretty well, and there were no major changes that broke *too* much for me at least, anything having to do with the stability of the system as a whole, and the general workflow with which I work with SW software, would be bad, to say the least.  9.5 has been a nightmare for a lot of people.  There has been more chatter about this release, than any other that I've seen, and a lot of the issues are egregious in terms of making it past the beta process in my personal opinion.

                                4)  While I think the slower release cycle is a great step in the right direction, I think you guys need to dedicate some resources to QA.  Whether that means a software testing platform or pushing your test build off of VM's onto hardware, with a considerable amount of devices to hit on a production instance that doesn't get wiped out completely after every test or not, is really up to you.  I think the SW test environment should contain a running instance of Orion that you actually use to monitor your own gear, along with the lab.  I know for a fact that you have support resources willing to take on some of that responsibility, should you guys be willing to put them behind it. 

                                5)  I'd suggest taking a look at http://www.aptest.com/resources.html to find the appropriate testing platforms and considering hiring a Director of QA who's responsible for testing, kicking the release back to dev, and re-testing until it passes ALL tests cleanly.  I cannot say for sure what happened with 9.5, but it seems like there wasn't really a valid process there.  If that requires increasing the annual maintenance costs somewhat, then so be it.

                                6)  Quite a few of the issues reported on thwack *seem* to be related to the installation process not being smart enough to account for the issues it might encounter in the wild.  Perhaps this is not the average experience, but there is again, an overwhelming amount of chatter about issues that sound like they shouldn't have made it out of QA.

                                There are a number of issues with performance of the web UI, and the fact that quite a large number of customers (I personally can't imagine why anyone would *prefer* to use the web ui) absolutely hate the fact that functionality has been removed from System Manager.  Perhaps the method is going to be to make a separate very fast web UI that is for administration only.  Personally, I'm not looking forward to being forced to use the web ui, and there are still issues like unmanaging nodes that I'm not sure have been resolved in 9.5 SP2.  I know all about tough choices, and putting resources where necessary, but overall, it seems like SW is being rather cavalier about significant usability and functionality loss.

                                I'm not trying to tell you per se, how to run your business, but having worked in more than a few software companies in my life, there are things that "we" do to ensure a good product and a happy customer.  At this point, I'm not sure I can honestly recommend upgrading until version 10, whenever that is released, and it's a drag, because I need the features in 9.5.  I need a better NCM, I need a better APM, I need a better NPM and Netflow.  I just can't deploy a possible better with the number of possible caveats.  I can't afford 2-3 days of working it out.  I need to roll an upgrade in an hour, and be done with it.  I have 500 people who would be blowing up my various phones and Inbox, were I to be as cavalier as some can afford to be.

                                Peter

                      • Re: Observation: General lack of testing environments

                        There is a big lack up understanding on the part of SW as far as licenses go. They will not issue a temporary license any more. They tell you to download the demo and do your testing there. Unfortunately the demo does not have the updated patches that you can install with the licensed version. You cannot install the patches to the demo to bring in the new fixes.

                        In my opinion it seems like SW doesn't care that these systems are critical in our environments and need to be tested out first.

                        I have always liked SW support but this process has been very frustrating.

                          • Re: Observation: General lack of testing environments
                            Miron

                            All,

                            My intention in starting this post was not to offer any critism to SW as im not in a position to comment on the effectiveness of the upgrade as I haven't proceeded yet myself.

                            It was more to draw the attention to the fact that Network Monitoring as a service has grown out of just being a nice to have and only used by techies to a complementary infrastructure management and enabler of business decision making tool. I am assuming and only based on some comments and personal experience that the users of the tool now include people like service desk staff, CTO, CIO, estates personel over and above the traditional techies.

                            As such the business should become more willing to supply the resources for development enviroments and yes at the same time an appropriate pricing structure for a dev license would ensure greater take up.

                            Regards

                             

                            Miron