13 Replies Latest reply on Mar 3, 2017 3:37 PM by jrahaeuser

    Upgraded server. Python API broken, PowerShell working

    toddtoddb

      Our server admin upgraded Solarwinds from a physical server to virtual server.  He said he did it by the book.  We have alot of python scripts that pull from the solarwinds API that are now broken.  When I download the SDK and try to run the example python script from the solarwinds server, I get the following:

       

      PS C:\> python .\SwisClient.py

      IP address of NPM Server:

      Username:

      Password:

      Invoke Test:

      Traceback (most recent call last):

        File ".\SwisClient.py", line 72, in <module>

          main()

        File ".\SwisClient.py", line 68, in main

          samplecode(npm_server,username,password)

        File ".\SwisClient.py", line 43, in samplecode

          aliases = swis.invoke("Metadata.Entity", "GetAliases", "SELECT B.Caption FROM Orion.Nodes B")

        File ".\SwisClient.py", line 18, in invoke

          return self._req("POST", "Invoke/%s/%s" % (entity, verb), args).json()

        File ".\SwisClient.py", line 37, in _req

          headers={'Content-Type': 'application/json'})

        File "C:\Anaconda\lib\site-packages\requests\api.py", line 50, in request

          response = session.request(method=method, url=url, **kwargs)

        File "C:\Anaconda\lib\site-packages\requests\sessions.py", line 468, in request

          resp = self.send(prep, **send_kwargs)

        File "C:\Anaconda\lib\site-packages\requests\sessions.py", line 576, in send

          r = adapter.send(request, **kwargs)

        File "C:\Anaconda\lib\site-packages\requests\adapters.py", line 412, in send

          raise ConnectionError(err, request=request)

      requests.exceptions.ConnectionError: ('Connection aborted.', error(10054, 'An existing connection was forcibly closed by

      the remote host'))

       

       

      I am attaching the script that I am running.  The only change was to enter a valid node ID on line 47 as requested by the script (changed to "NodeID=6"). 

       

      I worked with Solarwinds for a bit before they told me they don't support API.  I think this is more than a mere programing issue and something wrong with the internals of Solarwinds. 

       

      The odd thing is that I can create a basic powershell script and it runs fine.  Here is the powershell code I am running:

       

      Add-PSSnapin swissnapin

       

      $swis = Connect-Swis

       

      [array]$Nodes = Get-SwisData $swis 'SELECT NodeID, Caption, MachineType FROM Orion.Nodes'

       

       

      foreach ($Node in $Nodes)

      {

             if (($Node.Caption) -like "su*" ) {

      Write-Host $Node.NodeID $Node.Caption $Node.MachineType

             }

            

      }




      Working with Solarwinds for a short bit, we have already run the configuration wizard to have it fix any services or database issues.  I have restarted all services and still get the same error.  I have run the Solarwinds Diagnostics and searched for this error in the logs.  I see the same error code and response in the Core.BusinessLayer logs, but that doesn't tell me much. 


      Here are the versions we are running:  Orion Platform 2015.1.2, VNQM 4.2.2, IPAM 4.3, SAM 6.2.2, DPA 10.0.0, NCM 7.4, IVIM 2.1.0, QoE 2.0, NPM 11.5.2


      Any help is much appreciated.

        • Re: Upgraded server. Python API broken, PowerShell working
          tdanner

          My next step would be to take a look at the SWIS log file to see if it has any interesting errors. Run the script to trigger the error, then open C:\ProgramData\SolarWinds\InformationService\v3.0\Orion.InformationService.log and see if it has any clues.

            • Re: Upgraded server. Python API broken, PowerShell working
              toddtoddb

              This is the only thing added to the log after I ran the python query again:

               

               

              2015-11-10 13:16:43,563 [42] WARN  SolarWinds.InformationService.Core.InformationService - (null) Support! -- LONG RUNNING QUERY: Session uuid:dcfc09fb-7892-4ee0-bbce-98c7a9502689;id=706 - Query took 20540.0481 ms: SELECT Entity.Name, Entity.DisplayName, Entity.Description,

                              Entity.InstanceType, Entity.Namespace, Entity.FullName, Entity.BaseType,

                              Entity.IsAbstract, Entity.IsDynamic, Entity.Properties.IsKey,

                              Entity.Properties.Name, Entity.Properties.DisplayName,

                              Entity.Properties.Description, Entity.Properties.Type,

                              Entity.Properties.IsMetric, Entity.Properties.Units,

                              Entity.Properties.IsNavigable,

                              Entity.Antecedents.Name, Entity.Antecedents.SourceType, Entity.Antecedents.TargetType,

                              Entity.Antecedents.InstanceType,

                              Entity.Dependents.Name,

                              Entity.Dependents.SourceType, Entity.Dependents.TargetType,

                              Entity.Dependents.InstanceType, Entity.Antecedents.SourcePrimaryKeyNames, Entity.Antecedents.SourceForeignKeyNames,

                              Entity.Antecedents.TargetPrimaryKeyNames, Entity.Antecedents.TargetForeignKeyNames,

                              Entity.Antecedents.BaseType,

                              Entity.Dependents.SourcePrimaryKeyNames, Entity.Dependents.SourceForeignKeyNames,

                              Entity.Dependents.TargetPrimaryKeyNames,Entity.Dependents.TargetForeignKeyNames,

                              Entity.Dependents.BaseType, Entity.DisplayNamePlural, Entity.Aliases, IsNull(NodeChildStatusParticipation.Enabled, 'true') AS StatusEvaluationParticipationEnabled FROM Metadata.Entity  LEFT JOIN Orion.NodeChildStatusParticipation ON NodeChildStatusParticipation.EntityType = Entity.FullName

                                               AND NodeChildStatusParticipation.Installed = 'true'  WHERE Entity.Type ISA 'System.ManagedEntity' AND Entity.IsAbstract = 'false' AND Entity.FullName NOT LIKE 'Orion.VIM.%'  RETURN XML AUTO

                • Re: Upgraded server. Python API broken, PowerShell working
                  toddtoddb

                  I tried to run this query in the Solwardinds Database Manager, and I get the following error:

                   

                  Msg 4145, Level 15, State 1, Line 18

                  An expression of non-boolean type specified in a context where a condition is expected, near 'ISA'.

                  Msg 4145, Level 15, State 1, Line 18

                  An expression of non-boolean type specified in a context where a condition is expected, near 'ISA'.

                    • Re: Upgraded server. Python API broken, PowerShell working
                      tdanner

                      The Database Manager tool connect to SQL Server for running SQL queries. The query you have there is a SWQL query that needs to run against SWIS. You can do that in SWQL Studio.

                       

                      Since the python script didn't cause any errors to show up in the SWIS log, I suspect a connectivity issue. Let's see what happens if we make a request using curl.exe:

                       

                      curl.exe -k -v -u admin:password "https://localhost:17778/SolarWinds/InformationService/v3/Json/Query?query=SELECT+TOP+5+Caption+FROM+Orion.Nodes"

                       

                      Fix the "admin" and "password" to be a real username and password for your Orion install.

                       

                      I included the "-v" option to get verbose output.

                        • Re: Upgraded server. Python API broken, PowerShell working
                          toddtoddb

                          I ran the query through SWQL and got a response, although it looks like it took a while (17 seconds). 

                           

                          The curl request worked as well, and responded quickly:

                           

                          PS C:\Users\user\Downloads> curl.exe -k -v -u <username>:<password> "https://localhost:17778/SolarWinds/InformationService/v3/Json/Query?query=SELECT+TOP+5+Caption+FROM+Orion.Nodes"

                          * About to connect() to localhost port 17778 (#0)

                          *   Trying 127.0.0.1...

                          * connected

                          * Connected to localhost (127.0.0.1) port 17778 (#0)

                          * SSLv3, TLS handshake, Client hello (1):

                          * SSLv3, TLS handshake, Server hello (2):

                          * SSLv3, TLS handshake, CERT (11):

                          * SSLv3, TLS handshake, Server key exchange (12):

                          * SSLv3, TLS handshake, Server finished (14):

                          * SSLv3, TLS handshake, Client key exchange (16):

                          * SSLv3, TLS change cipher, Client hello (1):

                          * SSLv3, TLS handshake, Finished (20):

                          * SSLv3, TLS change cipher, Client hello (1):

                          * SSLv3, TLS handshake, Finished (20):

                          * SSL connection using ECDHE-RSA-AES256-SHA

                          * Server certificate:

                          *        subject: CN=SolarWinds-Orion

                          *        start date: 2010-05-28 11:28:51 GMT

                          *        expire date: 2039-12-31 23:59:59 GMT

                          *        issuer: CN=SolarWinds-Orion

                          *        SSL certificate verify result: self signed certificate (18), continuing anyway.

                          * Server auth using Basic with user '<username>'

                          > GET /SolarWinds/InformationService/v3/Json/Query?query=SELECT+TOP+5+Caption+FROM+Orion.Nodes HTTP/1.1

                          > Authorization: Basic bW1vc2hlcjE6R3VlcnJlcm8yNA==

                          > User-Agent: curl/7.28.1

                          > Host: localhost:17778

                          > Accept: */*

                          >

                          < HTTP/1.1 200 OK

                          < Content-Length: 176

                          < Content-Type: application/json

                          < Server: Microsoft-HTTPAPI/2.0

                          < Date: Tue, 10 Nov 2015 21:21:14 GMT

                          <

                          {"results":[{<valid data>}]}* Connection #0 to host localhost left intact

                          * Closing connection #0

                          * SSLv3, TLS alert, Client hello (1):

                            • Re: Upgraded server. Python API broken, PowerShell working
                              toddtoddb

                              Any idea why I can connect using curl and powershell API, but not the python API?

                               

                              Any help is appreciated.

                                • Re: Upgraded server. Python API broken, PowerShell working
                                  tdanner

                                  That's very strange. Some other things to try:

                                   

                                  - Turn on debugging for the python requests module - http://stackoverflow.com/a/16630836/13012

                                  - Run python on a different machine and connect remotely.

                                  - If you are in python2, try python3 and vice versa.

                                  - Use the Fiddler proxy to see the request from Python - Fiddler free web debugging proxy

                                  - Use Wireshark to see the request

                                    • Re: Upgraded server. Python API broken, PowerShell working
                                      toddtoddb

                                      python debugging:

                                       

                                      PS C:\Users\username\Downloads> python.exe .\SwisClient.py

                                      IP address of NPM Server:

                                      Username:

                                      Password:

                                      Invoke Test:

                                      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): <server>

                                      Traceback (most recent call last):

                                        File ".\SwisClient.py", line 104, in <module>

                                          main()

                                        File ".\SwisClient.py", line 100, in main

                                          samplecode(npm_server,username,password)

                                        File ".\SwisClient.py", line 75, in samplecode

                                          aliases = swis.invoke("Metadata.Entity", "GetAliases", "SELECT B.Caption FROM Orion.Nodes B")

                                        File ".\SwisClient.py", line 50, in invoke

                                          return self._req("POST", "Invoke/%s/%s" % (entity, verb), args).json()

                                        File ".\SwisClient.py", line 69, in _req

                                          headers={'Content-Type': 'application/json'})

                                        File "C:\tools\python2\lib\site-packages\requests\api.py", line 50, in request

                                          response = session.request(method=method, url=url, **kwargs)

                                        File "C:\tools\python2\lib\site-packages\requests\sessions.py", line 468, in request

                                          resp = self.send(prep, **send_kwargs)

                                        File "C:\tools\python2\lib\site-packages\requests\sessions.py", line 576, in send

                                          r = adapter.send(request, **kwargs)

                                        File "C:\tools\python2\lib\site-packages\requests\adapters.py", line 433, in send

                                          raise SSLError(e, request=request)

                                      requests.exceptions.SSLError: ("bad handshake: SysCallError(10054, 'WSAECONNRESET')",)

                                       

                                       

                                      Wireshark on the server shows the 3 way TCP handshake, then 1 packet of data, and then the solarwinds server sends a TCP reset.

                                       

                                       

                                      After alot of troubleshooting, I believe the problem is SSL handling by the requests module in Python.   I have 1 server that I was able to pull data from Solarwinds API through python, but I can't get it to work on any other host.  I have been trying a ton of stuff to get it to work, but no go. 

                                       

                                       

                                      Do you know if it is possible to make API calls over http, and remove SSL from the equation?  I have tried to change the following line:

                                      self.url = "https://%s:17778/SolarWinds/InformationService/v3/Json/" % (hostname)

                                      to

                                      self.url = "http://%s:17778/SolarWinds/InformationService/v3/Json/" % (hostname)

                                       

                                      But when I run the script, it sits idle for 120 seconds and the server sends a TCP reset.   Is there a setting in Solarwinds that allows the API to be accessed over HTTP?

                                       

                                      Let me know if anyone can troubleshoot the requests module as well.   I have 'verify=False' set in the request method, and I have added 'requests.packages.urllib3.disable_warnings()' to the script to no avail...

                        • Re: Upgraded server. Python API broken, PowerShell working
                          hmauricio

                          Were you able to fix this issue? I'm having the same problem

                          • Re: Upgraded server. Python API broken, PowerShell working
                            jrahaeuser

                            I ran into this issue, I had to force Python to use a specific SSL Protocol Version for it to work.