What a way to start a Monday.
We have a template for our Windows OSes that we use to gather some statistics via RPC. Most notably is Windows Process Queue Length that we use as part of the The Ultimate CPU Alert. Suddenly a bunch of nodes assigned to a single poller stopped being able to collect data via their assigned application monitors. We looked into all sorts of things -- network changes, account lock-outs, etc. Nothing stuck. It was a Windows 2008 R2 polling engine and it was polling a few subnets but it wasn't just a subnet issue as we had other polling engines polling devices on those subnets. The only common element was this Windows component via RPC.
Fortunately, we have a stellar Windows support team internally and our senior engineer grabbed onto the issue and she didn't let it go. May I present to you the root cause:
Patch Tuesday: KB3002657 Causing Authentication Problems with Exchange Other Apps | Patch Tuesday content from Windows I…
Once we manually backed out the change that KB3002657 made the template started succeeding when polling via RPC. Here's what our Windows team did on our polling engine. Note: These changes do NOT need to be made on the polling targets, just the polling engine.
Computer Configuration >> Windows Settings >> Local Polices >> Security Options >>Network Security: LAN Manager authentication level >> Send LM & NTLM responses
Though, if the word on the street is to be believed I would hold off on deploying this KB. ADs, Exchange and even EMC Isilon's are suffering issues related to this patch.
Practise safe patching folks.
