27 Replies Latest reply on Apr 2, 2018 4:35 AM by srikanth_hs

    SSH Connection issue from Solarwinds NCM Dashboard.

    srikanth_hs

      Hi Team,

       

      We have recently deployed NCM 7.7 and NPM 12.2 with latest hotfixes in our environment. We are experiencing connection issue while downloading configurations or accessing SSH only  from Dashboard. While using Putty from Server we are not experiencing this issue and this behavior is observed for most of the devices which is located in US office irrespective of manufacturer i.e. Cisco, Palo Alto and Aruba . This was not the case earlier during the time of POC but the version of NCM and NPM are not same during the time of POC.

       

      Even after the latest version of NCM 7.7 and NPM 12.2 it worked for few days and suddenly after some time we started facing this issue.

       

      We tried creating duplicate templates from Working device and only for the first time connection was success later all connections are getting failed. We have already opened support case about a month's back so far we did not get any fix or alternate option to continue with the scheduled backup. Support person has tried connection analyser tool to check the connectivity which is also getting failed.

       

      Not sure what could be the reason for this issue ?? Any suggestions/help is much appreciated.

       

      Note: Firewall rules are already in place and there is no block in any of the firewall. If there is any port or rule block in firewall even Putty session from the server should fail which is not the case here.

       

      Thanks and Regards,

      Srikanth

        • Re: SSH Connection issue from Solarwinds NCM Dashboard.
          bdufresne

          Srikanth,

           

          What is the error when trying to download the configs?  Are you able to download single configs from a specific node page?

           

          Bryan

            • Re: SSH Connection issue from Solarwinds NCM Dashboard.
              srikanth_hs

              Hi Bryan,

               

              Thank you for the reply. I'm getting Connection Timed Out error. Attaching the screenshot of one of the device for reference. I have created another template for ASA using working ASA template, initially it worked for the first time second time onwards i'm getting the same error. This issue is observed for majority of the devices. We have added 47 devices and scheduled Nightly Backup but unfortunately almost more than 50% of the devices are having this issue (close to 25 devices).

               

              Same template, same credentials (excluding enable password) and same model ASA is working absolutely no issues which is located in Bangalore office, but when i try this for my US office devices i'm facing this issue. I have already opened support case with Solarwinds Tech support (Case # 00053255) but so far no luck in finding out solution which is been almost a month now.

               

              Any help/guidance/inputs with steps is much appreciated. As we are very new to this product and this issue got escalated to our management now. I work during IST working hours i.e. Monday - Friday 9:30 AM to 5:30 PM IST.

               

                • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                  bdufresne

                  Srikanth,

                   

                  Do the credentials work through putty and other means?  Just checking to make sure the creds aren't locked out or enter incorrectly on SolarWinds.  Also, what version of SSH is specified in the ASA profile?  I would think it would be SSH2.  I would change the SSH auto to SSH2 or however it's configured just to test, perhaps the ASA is not allowing the SSH interrogation/negotiation.  Please let me know what works/doesn't work from here.

                   

                  Bryan

                • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                  sotherls

                  Bryan, let me add to this thread. We just upgraded to NPM 12.2 and NCM 7.7 last night.

                   

                  I noticed this morning that I had several NCM jobs showing 'Running 0%' and I opened a ticket for support as I could not figure out the root cause.

                   

                  While waiting, I also noticed that I was not able to SSH from Node Details to any device that usually worked. When I clicked on the SSH button it would bring up the SSH popup and prompted for login. I entered in the username and password (when prompted) but nothing happens - it just hangs. I then found that the SSH popup is not sending out any data (Wireshark shows no traffic from SSH).

                   

                  I then tried to Putty from the Orion server to various devices and they all work so it seems to be something within the SSH integration in NPM or NCM.

                    • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                      bdufresne

                      sotherls,

                       

                      It sounds to me like perhaps all the services did not come back properly?  I would run the Configuration wizard for the services and the website, and see if that resolves the issue.  If that doesn't work, there is a troubleshooter you can use called SolarWinds Active Diagnostics.  Give that a run through and see if that comes up with any issues/solutions.

                       

                      Bryan

                      • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                        srikanth_hs

                        Hi Sotherls/Bryan,

                         

                        First of all Thanks to Sotherls for recreating the issue and explaining everything in detail i.e. "

                        Bryan, let me add to this thread. We just upgraded to NPM 12.2 and NCM 7.7 last night.

                         

                        I noticed this morning that I had several NCM jobs showing 'Running 0%' and I opened a ticket for support as I could not figure out the root cause.

                         

                        While waiting, I also noticed that I was not able to SSH from Node Details to any device that usually worked. When I clicked on the SSH button it would bring up the SSH popup and prompted for login. I entered in the username and password (when prompted) but nothing happens - it just hangs. I then found that the SSH popup is not sending out any data (Wireshark shows no traffic from SSH).

                         

                        I then tried to Putty from the Orion server to various devices and they all work so it seems to be something within the SSH integration in NPM or NCM."

                        I guess there is no need to give further details about the issue .Sotherls was absolutely correct and whatever he experienced we are also experiencing the same behavior.

                         

                        Apart from that "Bryan, I ran the CW but still the SSH function is still not working. I am going to go ahead and open a ticket on it."

                        I have already tried performing this step while installing the latest Hotfix but still there was no luck, even support engineer was also on the WebEx session while performing this activity.

                         

                        Please let me know if you come out of any solution or fix for this issue. I will be awaiting for your response.

                         

                        Thanks a lot to both of you once again.

                         

                        Best Regards,

                        Srikanth

                          • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                            srikanth_hs

                            Hi Sotherls/Bryan,

                             

                            Any luck on finding out the solution for the reported issue ?

                             

                            Thanks and Regards,

                            Srikanth

                              • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                bdufresne

                                Srikanth,

                                 

                                Take a look at this and give the troubleshooting steps a try? NCM Jobs remain running and never complete - SolarWinds Worldwide, LLC. Help and Support

                                 

                                If that doesn't help you out, I would reach out to support, as I have not encountered this issue myself.

                                 

                                Bryan

                                  • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                    srikanth_hs

                                    Hi Bryan,

                                     

                                    Thank you for sharing the details. We will follow the troubleshooting steps and get back to you.

                                     

                                    Thanks and Regards,

                                    Srikanth

                                    • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                      srikanth_hs

                                      Hi Bryan,

                                       

                                      In the given link it clearly says this option is only for NCM 7.4 and we are using NCM 7.7 not sure if this will help us or not.

                                       

                                      Please suggest.

                                       

                                      Thanks and Regards,

                                      Srikanth

                                        • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                          bdufresne

                                          Srikanth,

                                           

                                          I understand it may not be a cookie cutter solution, but if you have any experience with troubleshooting, following these actions may lead you to the actual root problem or perhaps this fix will actually resolve your issue.

                                           

                                          The last resort fix method would be to backup the Orion database and re-install NCM, and if that doesn't work, entirely clean install the Orion platform and all of the modules you have licensed.

                                           

                                          Bryan

                                            • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                              srikanth_hs

                                              Hi Bryan,

                                               

                                              Thank you for clarifying on this. I tried this option still no luck.Just thought of updating on this point, I have tried capturing the traffic when initiated from NCM server and noticed that request from NCM Server dashboard is not sending on port 22. (I'm not sure if i have told you about this earlier). Apart from this while checking the connection from "Connection Test Analyser" tool located in "C:\Program Files (x86)\SolarWinds\Orion\NCM\Tools" below is the screenshot for the working and non working device behavior from Connection Analyser tool. Not sure if this information will help you in analyzing/identifying the problem.

                                               

                                              Please let me know if you need any more details on this issue.

                                               

                                               

                                               

                                              Thanks and Regards,

                                              Srikanth

                                                • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                                  bdufresne

                                                  Srikanth,

                                                   

                                                  Sorry, you made it sound like your issue was with NCM jobs, when really it's with connecting to devices, right?

                                                   

                                                  Can you connect to the non-working US devices through PuTTY on your workstation?
                                                  If yes, can you validate the snmp-server commands and the SNMP ACL are allowing the traffic to your SolarWinds?
                                                  Are there any overlapping ACLs that may be blocking this traffic out?
                                                  Are the non-wroking devices all at the same site?
                                                  Have you checked the firewall at this site?

                                                   

                                                  Please answer all the questions above, and let me know.

                                                   

                                                  Bryan

                                                    • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                                      srikanth_hs

                                                      Hi Bryan,

                                                       

                                                      Please find the answers inline.

                                                       

                                                      Sorry, you made it sound like your issue was with NCM jobs, when really it's with connecting to devices, right?

                                                      ---> It is actually both NCM jobs and as well as SSH connection from NCM.

                                                       

                                                      Can you connect to the non-working US devices through PuTTY on your workstation?

                                                      ---> Yes and it works fine without any issues.
                                                      If yes, can you validate the snmp-server commands and the SNMP ACL are allowing the traffic to your SolarWinds?

                                                      ---> Everything is in place. SNMP is working fine only SSH connection is failing/timing out.
                                                      Are there any overlapping ACLs that may be blocking this traffic out?

                                                      ---> No
                                                      Are the non-wroking devices all at the same site?

                                                      ---> Almost, some of them are located in different site within US.
                                                      Have you checked the firewall at this site?

                                                      ---> Yes checked the firewall and there is no issue if I establish Putty connection. Facing SSH connectivity issue only when initiated from NCM also downloading the configuration from NCM.

                                                       

                                                      I have strong doubt on the SSH client and its Cipher settings from the Solarwinds NCM. If there is an issue at the device it should block the connection from Putty as well but this is not the case here.

                                                       

                                                      Please let me know if you need any other details.

                                                       

                                                      Thanks and Regards,

                                                      Srikanth

                                                        • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                                          bdufresne

                                                          Srikanth,

                                                           

                                                          I would look into the egress firewall settings at your site and the ingress firewall settings at the remote site.  If NCM can SSH into anything, it's not an NCM issue.  It could be a credential issue, but it seems unlikely.  Since you can SSH in, but the server cannot, I think it is a firewall rule or ACL issue.  Drop PuTTY on the NCM server and try to SSH from the server desktop to one of the devices and let me know how that works.

                                                           

                                                          Bryan

                                                            • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                                              srikanth_hs

                                                              Hi Bryan,

                                                               

                                                              I have already tried that, I mean to say copied Putty on NCM Server and logging into the ASA from Server directly and this Putty session works absolutely fine without any issues. I'm facing problem only when I initiate SSH connection from Dashboard and also downloading the configuration from NCM Dashboard.

                                                               

                                                              If there is any firewall rule which is blocking this should block even Putty session as well from the NCM server but that is not the case here.

                                                               

                                                               

                                                               

                                                               

                                                              Thanks and Regards,

                                                              Srikanth

                                                                • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                                                  bdufresne

                                                                  Srikanth,

                                                                   

                                                                  Let me clarify again, as this isn't the issue that you had explained before..

                                                                   

                                                                  The issue is when clicking the SSH button on a Node's detail page?

                                                                   

                                                                  You mentioned downloading configs, when NCM does the config download job you are getting errors?  Can you copy/paste some of the errors from the Job error log for me to see?

                                                                   

                                                                  Have you checked the NCM credentials?  For the SSH button on the node details, is the IP being monitored the IP you would SSH into from your own box?

                                                                   

                                                                  Bryan

                                          • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                            sotherls

                                            Sorry for the late post to this.... vacation.

                                             

                                            Anyway, I opened a ticket for the NCM job issue - I simply had to disable then enable the jobs in question and then start them then they ran.

                                             

                                            On the SSH issue - it is still an unknown. Out of 4 servers upgraded I have only one not working correctly. The SSH login box pops up but I could put in bogus username/password and it never tries to contact the node (determined by wireshark as well as it never errors out). While not a show stopper since I put Putty on the Orion server and can connect that way it is frustrating.

                                             

                                            And the fact that half of SolarWinds support team thinks this is related to Engineers Toolset being installed (as this was the only way in earlier versions to be able to SSH from the Orion web GUI) doesn't help. Once I get past this argument we can start the troubleshooting.....

                                            • Re: SSH Connection issue from Solarwinds NCM Dashboard.
                                              srikanth_hs

                                              Hi Bryan/Sotherls,

                                               

                                              Just wanted to update you that we had support call with Cisco along with Solarwinds Support team, after capturing the traffic both from Putty Session and Solarwinds WOD SSH they noticed issue with WAN Optimized traffic when it is initiated from NCM WOD SSH Client. After bypassing the NCM Server from WAN Optimizer this issue got resolved.

                                               

                                              But yes what Sotherls is explaining was the exact same problem even we had.

                                               

                                              Thank you very much to both of you for simulating the issue and guiding me with the troubleshooting. You can consider this issue is resolved now. If required I will get in touch with you again.

                                               

                                              Once again Thanks a lot.

                                               

                                              Best Regards,

                                              Srikanth