Troubleshooting WMI account lockout issues - lessons learned

Good morning all,

We recently ran into a lot of issues with our WMI polling service accounts being locked out and I was hoping that I could help someone else in the future (or maybe myself) by listing a few of the things I eventually ran into.

1. Remember there are credential stores for products such as SRM, IPAM, UDT, VMAN, etc. in addition to the base Orion credential store.  If any accounts are used for duplicate polling and passwords change, they'll need to be changed within those products as well.

2. One of our lockout issues was caused by an engineer having used the WMI service account to log in to the Orion website over 2 years ago, presumably for some sort of testing.  I've found lockout events in Event Viewer (Security logs) in 2 places:

Event 4740 on the DC will give you the caller computer

Event 4625 on the Orion server where the account is locking out should be able to give you the caller process path.  Note: I've found that the security logs on our Orion server roll over pretty quickly, so this one is time sensitive.  You'll have to find it fairly soon after the lockout occurs.  Your mileage may vary!

In this particular instance, I found the caller process was SWISv3.  I combed the SWIS logs in ProgramData and found that something called Orion Account Validator was attempting to validate username/password for this WMI service account.  From there, I used Database Manager (or SSMS) to check out the Accounts table, where I found the particular service account with a last login of Feb. 2020.  I actually just deleted the row, unlocked the account, and have been trouble free since.

3. Another lockout issue on a separate Orion instance.  From event 4625 on the Orion server as mentioned above, I found the caller process to be Job Engine v2.  After much pain, log crawling, password resets, etc. I found that the issue was actually our Storage team having dropped this account from vSphere.  The password was correct, but since there were no permissions to log in or do anything, it was locking out this way. (VMAN)

Hope this is helpful to someone!

Parents
  • This has been very helpful thank you. I have found the 4740 event but the 4625 event on the polling engine has now given me C:\Program Files (x86)\Common Files\SolarWinds\JobEngine.v2\SWJobEngineWorker2.exe so i am now investigating your paragraph 3 VMAN. Funnily enough it was Virtulistation that highlighted the account locking :)

  • I am glad it helped!  It has definitely been a trying time to figure out where the lockouts are coming from.  I have had a couple of other issues since I made this post but they were both also the Job engine worker.  That can narrow it down a little, but not a lot.  I find myself staring at log files so often these days...  Best of luck!

Reply
  • I am glad it helped!  It has definitely been a trying time to figure out where the lockouts are coming from.  I have had a couple of other issues since I made this post but they were both also the Job engine worker.  That can narrow it down a little, but not a lot.  I find myself staring at log files so often these days...  Best of luck!

Children