ipmservice9.exe high cpu usage

jjohnson2

After adding a bunch (800+) of SNMP - User Experience monitors to monitor hardware on our Dell servers, the impservice9.exe program goes immediately to 99% cpu utilization and stays there for 5-10 minute spurts.  While it's at 99%, you pretty much can't do anything with ipMonitor.  It takes 2 or 3 minutes just to log in.  It's making the program pretty much unusable.  The program is installed on a Dell 2850 with 4GB of ram and 2 dual-core cpus so I wouldn't think it would be weak hardware.  Network use during the high cpu is minimal and only 371MB of physical RAM is being used.  Any ideas or suggestions?

jjohnson2

I got a call from Bryan with support and I believe we've gotten everything sorted out.  It seems that what was happening was that we had a bunch of custom snmp: user experience monitors set up on our security camera servers to monitor the Dell hardware (drives, controller, power supplies).  Each server had a total of 18 monitors.  They also had 2 network cards, one on a routable subnet and one that wasn't. 

When ipMonitor went to poll the machines, it performed a DNS lookup for the ip addresses because the monitors had been configured with the server names, not addresses.  One time, the address returned would be the routable ip.  The next it would get the non-routable.  When it got a non-routable, it would "hang" just waiting for a response or a timeout.  This would cause an alert to be thrown because as far as ipMonitor was concerned, the monitor was down.  According to support, when this happens, any group that that monitor is a member of then gets re-polled (or something along those lines) which caused even more CPU usage and delayed response.

The fix was that I went back and changed the configuration of the monitors to point to the routable IP addresses on the servers instead of the names of the servers.  MUCH better response time!

Still haven't been able to find the bounced emails but solarwinds.com has now been added to our whitelist so it shouldn't happen again.

 

Thanks to Bryan for all his help!

He also mentioned that there were a few other things that he'd seen cause high cpu usage.  The most common being a lot of event log monitors and huge event logs.  When the program has to parse 50,000 event log entries, it takes quite a bit of CPU.