6 Replies Latest reply on Jul 29, 2017 6:06 PM by lcsw2013

    Question regarding NCM

    lcsw2013

      Hi,

       

      I was wondering if any experienced problems when making a transition from an old deployment to a new deployment of solarwinds?

       

      We've almost ended up ditching solarwinds it's been such a headache.

       

      With NCM. We found that there is now way to properly transfer information over from one environment to the next. We were trying to transfer cred's and profiles as well and there was no way to get this done automatically meaning we had to manually input everything in the new environment. Having to perform things manually is not a problem if you have a small infrastructure. However, when you have elemental count of over 50k that's like trying to cram years worth of data into a new system in just a few days. Something is bound to go wrong. And in our case alot did go wrong. We had trouble adding devices to solarwinds. So alot of devices went in to monitoring without ncm. Then we had to manually add those. Plus manually add over a thousand more devices manually. And when we thought we had everything completed we notice that devices kept failing config backup's etc. Come to find out that the cred profile was not built in the new system. We called solarwinds to find there was no way to automatically transfer this over. We had to build it manually problem is that some of these profiles where built by folks who are either on vacation or are no longer with the company. Meaning that go live is further delayed having to wait on someone when it could of been avoided if solarwinds just made export and import easy.

       

      My question to the community is this. Have you ever had something similar happen? And how did you go about moving things over? Did you manually have to input everything or did you find a way to get this completed automatically?

       

      Any help is greatly appreciated.

        • Re: Question regarding NCM
          mesverrum

          To the best of my knowledge no one has come up with a way to automate the import/export process for credentials.  I run into it fairly frequently when I build new servers for clients.  Whenever I do a migration I make sure to screenshot the info I need regarding every NCM profile and recreate those in the new environment.  If you are comfortable in SQL you actually can copy everything from the Credentials and NCM_ConnectionProfiles tables over but the passwords won't work on the new system and each would need to be manually set within the UI. You definitely would need to do that right at the start of the new instance though because merging two credentials tables from live systems is likely to give you more headaches than you want to deal with.

           

          The method Solarwinds uses to encrypt the creds in the db does not allow you to copy them around or manipulate them and the dev team seems to keep pretty tight lipped about the encryption and have not built us a tool to automate moving them from system to system.  I can see why though, as having a tool to facilitate lifting all those admin type creds out of a Solarwinds instance would be extremely valuable for an unauthorized party and that's probably a liability they don't want to take on.

           

          If your organization has 50k elements and does not have a more structured scheme in place for distributing credentials to people who need them then I would say that in itself is the pain point you should be focusing on.  Don't feel bad, when I'm consulting I feel like 3/4 companies seem to have either this problem where nobody can track down passwords except the one person who is never around, or the alternative where everything is using the same simple passwords across the whole enterprise.  An Enterprise Password Manager would make it a non-issue to populate those passwords in the new system.  Getting all your passwords/community strings in place ahead of time is pretty much an absolute requirement before you even begin a monitoring platform migration, regardless of which one you are using.  Ditching NCM wouldn't change the fact that someone would still need to put those passwords into whatever tool you were to pick.

           

          One thing that might help you in the current situation is that once you finally do get all your credentials typed into the system go to the advanced settings of NCM and enable a setting where if a task fails due to a bad profile it will try all the other profiles to see if any of them work, this way it can automatically fix itself if you aren't sure which profile goes with which device:

          https://YOURSERVER/Orion/NCM/Admin/Settings/AdvancedSettings.aspx

           

          • Re: Question regarding NCM
            Craig Norborg

            Not really understanding your pain points, but I think its a lack of detail.   Most organizations I would think would use TACACS or radius for credentials and have a static profile set up for either all devices, or groups of devices, within Radius.    As mesverrum pointed out, you should be able to use "auto-detect" when adding devices and get them assigned the right profile that way.

             

            If an approach like this was taken, I'm curious what your pain points were.   

             

            If not using a centralized authentication server, I'd have to ask why not and what are you doing?

            • Re: Question regarding NCM
              HerrDoktor

              Credentials are the only thing that can't be transferred that easily. Everything else can be done via the API. I did this already with 3 of my clients.

              just like cnorborg mentioned, when you have a central monitoring credential set, the migration should be feasible.

              did you reach out to a SolarWinds consulting company for help?

              • Re: Question regarding NCM
                lcsw2013

                We tried to reach out to a consulting company but got voice mail. Left messages and no one returned for a long time. By the time they returned, through calls to solarwinds we ended up solving alot ourselves.

                 

                 

                I appreciate the help. When we were considering weather to upgrade in-place or build a new environment and move information over. Then we hit a snag when we had to move over more thank 600k ip's from one solarwinds to the other manually. We had to move cred's manually and had to go up the chain to get what we needed.

                 

                 

                Our calls to solarwinds, they made it seem to us like all this was easy no sweat. 1 2 3 and done. When really they could of just said you are better off doing an in-place upgrade to avoid the headaches. I actually escalated up the feedback chains within solarwinds that they need to work on getting customers like us better guidance and information.

                 

                 

                The headache came because we are in a pinch. Our temp licenses in our new deployment expire in 8 days. Upper management is not willing to buy or spend more money on lab licenses. Then when I have to return back to them saying these things are not possible and we need more time for manual effort this just makes solarwinds look very bad to management. They became frustrated with the software and company and threaten to find another solution.

                 

                 

                 

                I'm in constant meetings with higher up's daily having to provide progress reports among other things and it drains me when they as tasking me and telling me to get things done that basically has required me to work extremely long hours to get it accomplished.

                 

                 

                 

                This is why I came on here to see if anyone shared the same pains I've been going through and opening up feature requests to try and budge Dev's to do something about it.

                 

                 

                 

                I'm a certified Admin. The cert wasn't easy for me to acquire but I've been working with the software for many years. And felt it was a good way to further my knowledge. But these things caught me off guard. One thing I know for sure. I will never again build new. I will upgrade in-place and add on to current environment. Rebuilding only if we lose this environment. At least in-place upgrades just passes everything to the next version and I don't have to lose my mind with long hours trying to figure out ways  to make it all work.

                 

                 

                On a positive note for us. At the very least, this allowed us to build SOP's and further our internal documentations so that it helps us in future upgrades.

                 

                 

                Again, All replies where appreciated. And I thank you very much for the feedback. It was helpful.

                  • Re: Question regarding NCM
                    HerrDoktor

                    Don't be disappointed and don't throw away your option of building new.

                     

                    It really can help to improve performance when your system has been running for 10+ years and you have some "mishaps" in the Database. When you want to "migrate" again, give the right people a call. There are consultancy companies around that have the experience. Like I mentioned, I am migrating environments via the API not regularly but I have done it a few times. The scripts are getting better and better each time because you see new challenges and work your way around them in the API Scripts.

                      • Re: Question regarding NCM
                        lcsw2013

                        Sir, I understand and thank you for your help. I really do. Disappointment stems from distrust and people stepping over each other here and rejecting the truth. But that's a whole different story. My directors told me they called 2 consultancy companies at least twice without answer. They left a few messages but by the time they replied back to them, we had already figured it all out through tickets with solarwinds.

                         

                         

                        I agree options shouldn't be thrown away. But then again if you can't press one button and have a mirror image on the other side it's actually better to just start a new which was our plans till my network guys jumped in including their directors saying we need to transfer the data.

                         

                         

                        We tried ourselves to develope spreadsheets in terms of ipam and configure it for import on the other side. But it was actually taking much more time than just manually inputting the data in the system. So what we did is we spoke with out support leads and requested to have some assistance.  We had many folks jump in and start inputting data. That's how we cleared years of data in three days. We have over 606k ips in our ipam. For NCM it took a request from our VP to a network architect here and they gave us the cred's for NCM.

                         

                         

                        Lastly, You are right through experience things do become easier. It gave us a chance to future our documentations configure new sharpoint pages, etc. And reconfigure our SOP's. At the end it was pain. But next time we are better equipped. .

                         

                        Trust me when I say I understand and have a great appreciation for your feedback. Thank you very much