IPMonitor Service Stopping

I have recently upgrade our IPMonitor servers to version 11.1 from 11.0.1 however since the upgrade we are seeing the IPMonitor service randomly stop. When checking event viewer on all the boxes they show the same application error:

Faulting application name: ipmservice.exe, version: 11.1.0.789, time stamp: 0x5f88258a
Faulting module name: ipmservice.exe, version: 11.1.0.789, time stamp: 0x5f88258a
Exception code: 0xc0000409
Fault offset: 0x003b5cff
Faulting process id: 0x14bc
Faulting application start time: 0x01d6ed2e8fe258e4
Faulting application path: C:\Program Files (x86)\SolarWinds\ipMonitor\ipmservice.exe
Faulting module path: C:\Program Files (x86)\SolarWinds\ipMonitor\ipmservice.exe
Report Id: 74ea5b5e-8576-4b97-9c60-69aa0fcc97c6
Faulting package full name:
Faulting package-relative application ID:

I cannot find any other IPM posts with this error. I have had to restore one of the boxes back to 11.0.1 and now it is stable again but the others still randomly stopping.

Has anyone seen or have anyidea why this may be happening?

  • We have the same issue. We opened a ticket but support has not provided us with anything useful yet.

    This generally happens to us when we are making any changes in the GUI, i.e. Suspending monitors, running reports, editing users - but it is not consistent.

    If we aren't doing anything in the GUI, the system is stable.

    What we have done so far is:

    - Create additional exceptions in Symantec Endpoint Protection and Carbon Black Defense

    - Re-Installed the update

    But these did not help.

    Have you had any luck with yours? Have you opened a ticket?

  • Any luck or resolution from support on this?

  • We downgraded to ipMonitor Version: 11.0.1 build 136 and have had no issues since - support didn't have a better answer.

  • Yesterday this started happening to me, running 11.1.1 build 980

    After posting, I realized this started happening right around the time I added a credential for an SNMPv3 monitor yesterday. I completed the steps below and finally can got service to stay up for longer than five minutes, 30 minutes now.

    My SNMPv3 username has a dash in it and my passwords are 16 random characters, not sure if that has anything to do with it but I'll try to add the monitor and credential back in later.

    1. C:\Program Files (x86)\SolarWinds\ipMonitor\config\cred
      1. changed the *.x file modified yesterday to *.old
    2. C:\Program Files (x86)\SolarWinds\ipMonitor\config\monitors
      1. changed the *.x file modified from yesterday to *.old
    3. Restarted all ipMonitor services 

    I'll update tomorrow if this was the real fix for my issue.

  • Thanks for the information, hoping this works out for you

  • Has this issue ever been resolved for you?  Im having the same issue and have been since upgrading to 11.1.1.980.  It crashes several times a day while I'm in the UI for no reason at all.  I haven' been able to find any entry in the logs other than the application log.  I've been running IPM in debug logging mode in hopes to find something, but got nothing.

  • Let us know what support says - we were never able to get any logs for this issue.

  • I did finally catch it logging errors when it failed while I was updating monitors:

    [2022-02-15 10:18:33.175] [TID 6160] [error] Unhandled exception (C:\buildAgent\work\b671993bbbec825b\Src\MainService\TemplateServices\variable_providers.cpp:2117): bad allocation
    [2022-02-15 10:18:33.184] [TID 6160] [debug] wsTcp::Reset::shutdown error=10038 (socket 6732)
    [2022-02-15 10:18:33.184] [TID 6160] [debug] wsTcp::Reset::shutdown error=10038 (socket 6732)
    [2022-02-15 10:18:33.201] [TID 1060] [error] Unhandled exception (C:\buildAgent\work\b671993bbbec825b\Src\MainService\TemplateServices\variable_providers.cpp:2117): bad allocation
    [2022-02-15 10:18:33.480] [TID 6004] [debug] THOLD(1921479729131): comparing thresholds. lv_max=90.000000 value=4.073858 bPassed=1 bOk=1 bWantGT=0 datakind=80001
    [2022-02-15 10:18:38.880] [TID 7880] [info] ipm_snap::SnapResultsListenerGRPC::PostResults - received message with 1 metrics
    [2022-02-15 10:18:38.880] [TID 7880] [info] ipm_snap::SnapResultsManager::PostResults - received results for task [1775437202401]
    [2022-02-15 10:18:40.040] [TID 1060] [error] Unhandled exception (C:\buildAgent\work\b671993bbbec825b\Src\MainService\TemplateServices\variable_providers.cpp:2117): bad allocation

  • For those who are still needing help with this, I got the root cause for it from support:

    1) On the ipMonitor interface, navigate to the Dashboard and delete the "Do more with IP monitor" widget
        (do not use this widget anymore)

    2) Log in to your ipMonitor server and stop the following services:

    ipMonitor Reporting Service
    ipMonitor Service
    ipMonitor Snap Telemetry Framework
    SolarWinds Discovery Service

    3)  Once the services are stopped, navigate to the following folder   

    C:\Program Files (x86)\SolarWinds\ipMonitor\config\netscan

    4) Move the contents of this folder to a backup folder, preferably in the Desktop

    5)  Move the ipmonitor.dat file if it exists, located at C:\ProgramData\SolarWinds\DiscoveryEngine folder to another backup folder in the Desktop

    6) Restart the following services again:

    ipMonitor Reporting Service
    ipMonitor Service
    ipMonitor Snap Telemetry Framework

    Please do note that the widget will get restored after you log out and log back in. This is not a cause for concern.

    -------

    I took these steps and my 11.1.1.980 is stable now.  In my case, it was an upgrade from a previous version and netscan files from previous versions are not compatible with the current version and had to be removed.

    I really hope this helps others resolve their issue with random crashes.  Mine hasn't crashed in 24 hours now and I've been working it it for a few hours updating some monitors and it would crash within 5 minutes of working in it.  Super stable now!

    Thanks SolarWinds Support!