6 Replies Latest reply on May 6, 2016 11:37 AM by netlogix

    Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)

    netlogix

      Anybody have having/had issue polling their vCenter?  I opened a case (963304) with support three weeks ago, but they haven't made any progress and are trying to get me to adjust my retention.

       

      We removed and re-added one of my vCenters, and it won't list any host under it.

      If I add an vSphere host itself, it won't list any guests.

      We tried multiple different user accounts, all test out fine, but no matter how long we leave it, the guests are not listed.

      We used a web browser to look at the MOB directly and it works.

      Virtualization Manager is able to poll it without any issue (for testing, we removed the integration)

       

      Here is what I am running: Orion Platform 2015.1.3, WPM 2.2.1, NCM 7.4, NPM 11.5.3, DPA 10.0.1, QoE 2.0, IVIM 2.1.1, SAM 6.2.3

       

      I have re-installed (and repaired, and marked for compatibility):

      SolarWinds Integrated Virtual Infrastructure Monitor

      JobEngine

      JobEngine.v2

      .Net 4.5

       

      Upgraded

        NPM 11.5.3 (caused me to need a buddy drop because I could not list resources after that)

        APM 6.2.3

        IVIM 2.1.1 (I download it, but one of the previous upgrades did it for me)

       

      I don't know what else to do, an suggestions?

       

      Hey, aLTeReGo, if this is the wrong place, can you move it (or tell me where/who is the PM, or have a moderator do it)? Thank you!

        • Re: Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)
          lukas.cerny

          Hi, this looks like a known issue. In those versions VMware changed number of allowed elements in queries. This happens only when the polling is done through Orion, Virtualization Manager doesn't have this issue. There are actually two workarounds which you can try and might be able to help.

           

          1. There is a KB article by VMware which describes setting that can be changed to increase number of allowed elements in queries: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2107096
          2. You can modify a configuration file on SolarWinds side, this will cause that we will execute more smaller queries. To do so:
            1. Backup and modify SolarWinds.VIM.Poller.dll.config (by default in: c:\Program Files (x86)\SolarWinds\Orion\VIM\SolarWinds.VIM.Pollers.dll.config)
            2. Change the value of VMwarePerformanceCounter.ChunkSize to something smaller (for example 3, but you will need to experiment a bit with what should be the correct value)
            3. After you save the change restart Orion services

           

          Hopefully one of these should help.

          1 of 1 people found this helpful
            • Re: Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)
              netlogix

              Changed the vmware side to 128 (was at 64), and Check Orion and it is/was set at 30.  The article also said there would be errors about the query size being rejected in the vpxd log but there aren't any.

               

              I was hoping that was going to be it.  Plus the test server I am pointing at is only two hosts and ten guests.

                • Re: Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)
                  lukas.cerny

                  Strange, can you check logs in c:\ProgramData\SolarWinds\Logs\VIM\Jobs for following: System.Web.Services.Protocols.SoapException: This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'.? In case you can see such errors there, it should be caused by this query metrics limitation. If that is the case, try to decrease the chunk size. The chunk size is not equivalent to the amount of metrics used, so even if chunk size is 30 it can still try to use more than 128 query metrics (so I'd recommend to start with small number an then you can try increase it step by step to see where is the threshold for your environment).

                   

                  If you don't see the issue there, please open a case with support.

                    • Re: Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)
                      netlogix

                      there are no "This operation is restricted by the administrator", but there are:

                       

                      System.Web.Services.Protocols.SoapException: A specified parameter was not correct.

                      System.Web.Services.Protocols.SoapException: A general system error occurred: Operation was canceled

                      System.Web.Services.Protocols.SoapException: A general system error occurred: An existing connection was forcibly closed by the remote host

                        • Re: Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)
                          netlogix

                          I changed the VMware values to unlimited, but it still is not working.

                           

                          I found a another error message as I was looking around:

                          "Error while connecting to VMware device - The request failed with HTTP status 503: Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:00000000006816D0] _serverNamespace = /sdk _isRedirect = false _port = 8085)."

                           

                          One web search turned up "make sure vpxd is listening" and using netstat I found:

                          Proto  Local Address          Foreign Address        State

                          TCP    0.0.0.0:80             0.0.0.0:0              LISTENING       2180

                          TCP    0.0.0.0:443            0.0.0.0:0              LISTENING       2180

                          TCP    127.0.0.1:8085         0.0.0.0:0              LISTENING       5364

                           

                          Image Name                     PID Session Name        Session#    Mem Usage

                          rhttpproxy.exe                2180 Services                   0     17,244 K

                          vpxd.exe                      5364 Services                   0    953,052 K

                           

                          I can manually go to https://x.x.x.x/mob and move around.

                           

                          Any other ideas?

                  • Re: Unable to poll VMware (it was after 5.5U3 on some and 6.0U1)
                    netlogix

                    Thank you support!!!

                     

                    So it wasn't the upgrade, but it was that a proxy got configured (I'm not sure how):

                    Disable Default Proxy Server for Orion Virtualization Monitor Connections to VMware vCenter

                     

                    To check to see if a proxy is configured:

                    Netsh Winhttp show proxy

                     

                    I probably could have just removed the proxy too:

                    Netsh Winhttp reset proxy

                     

                    We also had some odd remnants hanging around in some of the VIM_ tables we had to remove.

                     

                    Thank you Michael!

                    1 of 1 people found this helpful