17 Replies Latest reply on Feb 9, 2018 5:25 AM by uniswtc

    Windows Services Template (SAM) shows could not connect for WMI access

    uniswtc

      Hi All,

       

      We are in a different situation. We have been using Windows Generic services template on all windows servers. But, some of the servers are failing with the below error, when tested against the components.

      The error is: "Test failed with "Unknown" status on <ServerName>. Unable to connect to <IpAddress> for WMI access".

       

      When checked with Server team, they had confirmed that all the servers has the same admin privileges for the account being used to connect through WMI.

      Because of this the server shows the status of the server as "One or more application monitors is in Unknown state."

       

      Hence, I would like to know what could be the reason, when the same account being used for all the windows servers, with same admin privileges. Where to check?

      We need to fix the issue asap.

       

      Thanks,

      Naren.

        • Re: Windows Services Template (SAM) shows could not connect for WMI access
          mesverrum

          Use the Microsoft WBEMTest tool to test and see what you get when you try to connect to the server via WMI.

           

          Testing WMI Connectivity - SolarWinds Worldwide, LLC. Help and Support

           

          Beyond confirming if there really is a problem with WMI, you should just troubleshoot it as per MS docs, WMI Troubleshooting (Windows)

           

          If WMI tests fine it would probably indicate a problem with your polling engine and you may need to open a ticket with support.

          1 of 1 people found this helpful
            • Re: Windows Services Template (SAM) shows could not connect for WMI access
              uniswtc

              Hi mesverrum,

               

              Thank you for the details. I did run the wbemtest and got the Access Denied when I rant the tool from Solarwinds App Server. I tested the same for another server, where we did not see any errors, it has shown the details when clicked on Enum Classes.

               

              Does it mean, the admin access to the Solarwinds admin account (which is being used to connect to the windows servers being monitored) differently for domain controllers? The one which I had tested is a domain controller.

                • Re: Windows Services Template (SAM) shows could not connect for WMI access
                  mesverrum

                  Hard for me to speculate, the specific quirks of Windows Domain account permissions have always been a little out of my wheel house (network engineer before I took up this career of Solarwinds trainer).  In this case you do have a bit of a smoking gun at least that there is something different between how that account is being treated by your servers, i'd spot check the group memberships and maybe ask about the GPO's in effect in the different areas to be sure that they actually match up between the box that works and the box that doesn't.  All servers are members of the same domain I'm assuming?  Cross domain trust has occasionally been a sticking point at places where I've worked.

                   

                  I like using WBEMtest to test these things out because it keeps the Windows guys from blowing me off with any "everything is fine, your tool just doesn't work" nonsense.

                  • Re: Windows Services Template (SAM) shows could not connect for WMI access
                    leigham martin

                    Hello,

                     

                    The servers that SolarWinds are failing to connect to, are they all DC's by any chance?

                     

                    Regards,

                     

                    L.

                      • Re: Windows Services Template (SAM) shows could not connect for WMI access
                        leigham martin

                        Hello,

                         

                         

                         

                        Sorry slightly confused, you say you dont use WMI at all due to the weight, but then the error message your getting relates to WMI?

                         

                        Do you guys work for the same company?

                         

                        Has all your SNMP config been done on all your servers, when you test the SNMP credentials in SAM does it come back ok?

                         

                        What size is your environment? We run WMI across the estate of around 100-150 physical servers and 350-400 vm's and we dont have any issues with weighting or network traffic, its busy yeah. but its ok.

                         

                        Do you have an additional poller or is everything being done via 1 single Orion server?

                         

                        If you are using a second poller you wouldn't setup the monitoring on that site, you would do it from your master install and just select the additional poller to do the work for you.

                         

                        Are your servers that are failing on a domain or are they local workgroup (DMZ env)? Are they behind firewalls or are they all on the same network?

                         

                         

                         

                        Thanks

                         

                         

                         

                        L.

                          • Re: Windows Services Template (SAM) shows could not connect for WMI access
                            uniswtc

                            Hi leigham martin,

                             

                            Sorry, I could not respond quickly, due to busy with other work. For your first question, yes, majority of the servers are DCs. But, we are getting the error for other type of servers as well.

                             

                            sbox1107 is not from our company. Please see below the answers inline to your queries.

                             

                            Has all your SNMP config been done on all your servers, when you test the SNMP credentials in SAM does it come back ok? - Yes

                             

                            What size is your environment? We have 1400+ windows servers (around 40% are DCs), 200+ VMs we do have. Including network devices we have 4k+ devices in SW env.

                             

                            We run WMI across the estate of around 100-150 physical servers and 350-400 vm's and we dont have any issues with weighting or network traffic, its busy yeah. but its ok. - We have enabled the generic services template for all the windows servers using WMI with credentials.

                             

                            Do you have an additional poller or is everything being done via 1 single Orion server? Yes, we do have an additional poller.

                             

                            If you are using a second poller you wouldn't setup the monitoring on that site, you would do it from your master install and just select the additional poller to do the work for you. - We had enabled monitoring on additional polling for all DC devices only. Rest all being monitored byt he primary poller only.

                             

                            Are your servers that are failing on a domain or are they local workgroup (DMZ env)? Are they behind firewalls or are they all on the same network?  As updated above, some of the failing servers are Domain Controllers. They are not behind a firewall, in the same network and allowed from Solarwinds server.

                             

                            We did also checked for the errors by running WBEMTEST tool and got access denied with error as "0x80070005".

                             

                            Hope I have provided all the required information. I would wait for your response/update with a solution . Thank you so much for your time.

                              • Re: Windows Services Template (SAM) shows could not connect for WMI access
                                leigham martin

                                uniswtc

                                 

                                Im assuming your using some kind of service account or a 'Master' account which has access to all your servers? - is this a domain admin account?

                                 

                                Well, least we ruled out any network/firewall connection issues if your on the same network with nothing in between! -

                                Question... why did you choose to use agents to carry out the WMI monitoring when you could do that directly from the poller (Saving the need for agents)? - Theres nothing wrong with that, its just an extra step i guess.

                                 

                                Are all your Windows Firewalls turned off?

                                 

                                When you poll the servers in question can you check the following location/log files for any errors?

                                 

                                C:\Program Data\SolarWinds\Logs\Agent\

                                 

                                There should be 2 different log files (Agent Service & Agent Discovery) check them both, see if you see any errors in there like the one your getting when doing the WBEMTEST process.

                                If there are errors, post them back here if you dont mind? - obviously redacting any server names or IP addresses.

                                 

                                If your seeing the error that you get from WBEMTEST in the log file then there could be an issue with the Windows Management Instrumentation Service local to the server itself?

                                Have a look at the below Technet article, somebody with a similar issue to yourself in regards to the WBEMTEST

                                 

                                Remote WMI Access denied Error - 0x80070005

                                 

                                If that looks ok then you could try restarting your Windows Management Instrumentation Service on the affected servers.

                                 

                                One thing to also check is that you use the same generic WMI template across all your servers, go into a node that is not working, the green spot that points to your generic services application, click into it so it takes you to the page with all the stats and click edit application monitor.

                                Then, click the little arrow next to 'Advanced' so that it drops the menu down, make sure that preferred polling method is 'Agent' as seen below

                                 

                                Please let me know if any of the above helps.

                                Thanks

                                 

                                  • Re: Windows Services Template (SAM) shows could not connect for WMI access
                                    uniswtc

                                    Hi leigham martin, thank you for the quick response . Much appreciated.

                                     

                                    Yes, we have a domain admin service account for Solarwinds and which is being used and added on the server administrators group as well. The same account has been provided for WMI access as well. Out of 1400 servers, getting mentioned access error for 300+ servers only.

                                     

                                    BTW, we do not have agent enabled monitoring. All are agentless monitoring only with SNMPv2c (few are with ICMP only as well).

                                     

                                    For the remaining questions, I will check and provide the answers and screenshots (of course blackout the server or IP address ) by tomorrow.

                                      • Re: Windows Services Template (SAM) shows could not connect for WMI access
                                        leigham martin

                                        Hi uniswtc

                                         

                                        Apologies, it was a long day yesterday, not sure where i got the idea about agents, i just read back through and you didn't mention it!

                                         

                                        If your not using agents on the affected servers then i believe those log files wont be there, however, you could check on your SolarWinds logs here: C:\ProgramData\Solarwinds\Logs

                                        To see if there are any exceptions being thrown out.

                                         

                                        As a side note, have you logged this with support also? I will help as much as i can based from my own experiences and issues but of course they are the experts.

                                         

                                        I believe we can rule out SNMP here as your Generic services template is WMI and thats where its failing.

                                        What version is your SAM installation? Are you on the latest?

                                         

                                        Still think its definitely worth restarting the WMI service on at least 1 affected node to see if that sorts your issue, it has to be something bug related because if its a domain admin account your using and its throwing out Remote WMI access denied error messages, well.. that doesn't add up!

                                         

                                        When you tried the WBEMTEST tool were you using the same account?

                                         

                                         

                                        Regards,

                                         

                                        L.

                                        1 of 1 people found this helpful
                                          • Re: Windows Services Template (SAM) shows could not connect for WMI access
                                            uniswtc

                                            Hi leigham martin,

                                             

                                            Sorry, I could not respond immediately. I have searched the files under Logs directory as mentioned by you with "Access", "Error", "unable", etc. But could not find anything. The directories are as shown below. Please advise, any particular directory need to check for any particular words.

                                            We have not raised a case with Support. I am planning to do. I will wait for your last response and then go for it.

                                            We are having SAM 6.2.2. We are upgrading to the latest next weekend (2/10).

                                            I have requested to restart the WMI service on one server. Tomorrow, they would do that (hopefully) and update you on the result.

                                            Yes, I ran the WBEMTEST using the admin service account only.

                                              • Re: Windows Services Template (SAM) shows could not connect for WMI access
                                                leigham martin

                                                Hi uniswtc

                                                 

                                                You can look at the following log files;

                                                 

                                                There is a lot of noise generated in these log files here but they are for all your SAM Monitors, id advise opening them with Notepad ++ as its easier to navigate through using that.

                                                Once you've opened it in Notepad++ do a CTRL+F to find and look for either the IP address or the host name of the server your having issues with, i imagine there is probably an exception for it in one of those logs.

                                                 

                                                If you wouldn't mind, can you post the exception? I advise to redact the IP or Host name so that you avoid letting people know your server info!

                                                 

                                                I would highly recommend that you move to the latest Orion and SAM versions, you are quite a few versions out of date and this isnt going to help any situation, plus there a lots more features in the latter versions to what you are currently on.

                                                Have they managed to restart the Windows Management Instrumentation Service on the problem server in question? has that solved your issue?

                                                 

                                                Regards,

                                                 

                                                L.

                                                1 of 1 people found this helpful
                              • Re: Windows Services Template (SAM) shows could not connect for WMI access
                                sbox1107

                                Have you run winrm quickconfig on the server?

                                 

                                If the account is a local administrator on the remote server, and if so, are you able to login to the remote server and manage services?

                                 

                                If the account is not an a local administrator you will have to grant permissions to the service manager for standard services and any non-standard services individually.  See: How to create a non-administrator user for SAM polling - SolarWinds Worldwide, LLC. Help and Support for details.  The account requires specific privileges on the Win32_Services object and the article above shows how to do it.

                                 

                                If that fails, try changing the polling method from WMI to RPC in the monitor template.

                                 

                                WMI or RPC for service polling