32 Replies Latest reply on Aug 28, 2018 7:19 AM by jm_sysadmin

    IPAM Scan job Status question

    scotthill

      In IPAM in the scan job status in IPAM settings, the status is "queued" and never changes. Scan trigger is automated. Last discovery shows the date probably when it was first set up.

       

       

      When you chose a subnet and click on the Scan feature, it doesnt scan, it appears as if it just Qued up the scan job but wont ever run. Looks like there is multiple jobs in there as well. 

       

       

      Is that normal? Is that because they were queued up run automatically and never ran? How do you kick them off and/or remove them if needed?

       

       

      Orion Platform 2015.1.0, SAM 6.2.0, QoE 2.0, IPAM 4.3, NPM 11.5, PM 2.0, Storage 5.1.0, IVIM 2.0.0

        • Re: IPAM Scan job Status question
          sathya4046

          Could you please check your Orion server date and time? Please sync it to current date & time, if it is different.

          • Re: IPAM Scan job Status question
            orionfan

            Scotthill,

             

            Did you ever fix this?  I just discovered I have many scan jobs (8 pages) that are in a queued status for over a month.

            • Re: IPAM Scan job Status question
              orionfan

              I have opened a case on this.  I also found this in their KB:  SolarWinds Knowledge Base :: IPAM Scans Not Running

              • Re: IPAM Scan job Status question
                orionfan

                This is what fixed the issue for me.  These instructions may be unique to me because we have other Solarwinds products installed.  The Jonengine.msi installation failed initially due to UAC.  I had to run an elevated command prompt and then run the msi which worked.

                 

                1. Stop all SW Orion services using the Orion service manager located at start, all programs, solarwinds orion, advanced features.
                2. Browse to the Path:
                  C:\ProgramData\Solarwinds\Installers
                3. Right click on Jobengine, Jobengine.v2.msi and the CollectorInstaller.msi and uninstall each.
                4. After they are all removed, run them again to reinstall those 3 files, if get a prompt, choose Typical install.
                5. Restart all SW Orion services

                *The User Access Control (UAC) on the Windows Server may need to be disabled and the server needs to be restarted.



                  • Re: IPAM Scan job Status question
                    jpashton

                    I've been fighting this for a few months now with mixed results.  I tried the fix of re-installing the Jobengine, Jobengine.v2, and the CollectorInstaller after opening a support case and getting the response to perform that.  Initially, this worked, but after two days all my scans are back to queued again.  This is very frustrating to say the least.  Nothing seems to work for very long with this issue.

                      • Re: IPAM Scan job Status question
                        cbunge

                        My scans worked until adding in DHCP servers.  Bot the jobengine installers and file replacement fixes are temporary only.  It's almost as if the scanning locks on one of the DHCP servers and never allows the next batch to run.

                         

                        Items tried:

                        Adjusted number of concurrent scans - No Impact

                        Adjusted transient period - No Impact

                        Adjusted with and without SNMP - No Impact

                        Removed down DHCP servers - No Impact

                        Replaced Jobengine v1, v2, and agent files - Fixes scan for a scan or two

                        Reinstalled jobengines - Fixes scan for a scan or two

                         

                         

                        We don't have that many subnets monitored at this time either.

                    • Re: IPAM Scan job Status question
                      c4ucka54m0r3

                      I've just found out I am having this issue as well.

                        • Re: IPAM Scan job Status question
                          cbunge

                          Currently I have a long going support ticket with tech support.  I'll update if I get anything, but right now anything they have me try and do temp fixes at most for a day.

                           

                          I did however have an issue with my additional web pollers getting a memory leak issue and they had a custom patch for my businesslayer.dll so maybe this could also be a point to look at within IPAM.

                        • Re: IPAM Scan job Status question
                          jm_sysadmin

                          It happens to us more than we would like. I built a SQL Server User Experience Monitor in SAM to pull data on how old our scans are. The SQL is below, since our scans run every 4 hours, we shouldn't have any over 5 hours. If that component fails we send an alert and like mentioned in the link above, replace JobEngine35.sdf

                           

                          SELECT Count(*)

                            FROM IPAM_ScanInstance

                            where (DateDiff(day, queuetimestamp, getdate()) > 0) or (DateDiff(hour, queuetimestamp, getdate()) >5)

                          • Re: IPAM Scan job Status question
                            jpashton

                            After a lengthy support case with SW, I have had scanning up and going for a week now.  I don't know which combination of fixes truly helped, but I essentially ended up fully re-installing IPAM program files.  All of the settings are kept in the database, but what I ended up doing was completely removing IPAM from the Add or Remove Programs and then deleted all of the IPAM related folders in the Program Files area and inetpub folder.  After rebooting, I then followed the steps to install IPAM.  This has been going for a week now and so far scanning is still running. 

                            • Re: IPAM Scan job Status question
                              shack

                              I absolutely despise the some of the solutions provided. Your fix is "Do a complete reinstall" of the program.  That is llike saying reinstall windows for a printer issue.  I'm in the same boat here guys.  I have hundreds of subnets queued up and most of them haven't scanned since 10/25/2015. 

                                • Re: IPAM Scan job Status question
                                  jm_sysadmin

                                  I won't argue that its not good,  I can say that the reinstall did improve reliability, but its not completely eliminated. The reinstalls are not something we do frequently (though many product updates have us rerun the config wizard which can reinstall), we stop the services and recreate JobEngine35.sdf then restart them unless the issue happens multiple times in a week. Currently it happens once or twice a month, and we fix it quickly with my alert list above. We also moved our scans from 4 hours to 8, and adjusted the alert accordingly.

                                   

                                  Ideally IPAM gets shown some update love in the near future.

                                    • Re: IPAM Scan job Status question
                                      Impaler

                                      This is ridiculous and totally unacceptable. I don't have the time, nor should I ever have to do a complete reinstall, much less to this periodically every time the scans seize. I've been periodically doing the .msi uninstall/reinstall routine when necessary, but I'm having to do this more and more frequently. Please fix this SW....... 

                                  • Re: IPAM Scan job Status question
                                    jhciotti

                                    For all the sufferers: Improved IPAM Scan Job Management

                                    1 of 1 people found this helpful
                                    • Re: IPAM Scan job Status question
                                      orionfan

                                      Good idea on the feature request.  I have spent quite a bit of time working with support on this.  The last time it failed, they had me do the following.  Also, he saw in the diagnostics that there were some indexes that were highly fragmented and suggested I have my DBA's rebuild those indexes.

                                       

                                      Process to force the business layer (module engine) to load plugins in separate processes (32 bit):

                                      Update this file:

                                      C:\Program Files (x86)\SolarWinds\Orion\SolarWinds.BusinessLayerHost.exe.config

                                       

                                      Set this line like so:

                                      <add key="ForcePluginsInSeparateProcess" value="True"/>

                                       

                                      Save and restart the Module Engine service.

                                       

                                       

                                      The 64 bit process is more in depth. I would recommend trying the separate process steps for 32 bit above first and see how it performs. If they still crash, then try the process below.

                                      First you modify the config file for the specific module, for example UDT:

                                      C:\Program Files (x86)\SolarWinds\Orion\UDT\SolarWinds.UDT.BusinessLayer.dll.config

                                       

                                      Set this line like so:

                                      <SolarWinds.BusinessLayerHost LoadPlugins="true" SeparateProcess="true" Support64="true" />

                                      1 of 1 people found this helpful
                                      • Re: IPAM Scan job Status question
                                        Kurt H

                                        So does anyone know if there is any word from SolarWinds on if they are going to come out with a hotfix to resolve this problem with IPAM. We ran into the problem again where all our scan jobs just locked up, and showed next scan ASAP. Of course supports only answer is delete the job engines and create a new one. This is annoying that this software is not getting the attention or this problem is not getting fixed when so many people are running into it. How many times do we have to contact support and put in support tickets before someone fixes the problem??????

                                          • Re: IPAM Scan job Status question
                                            scotthill

                                            Our IPAM quit scanning again. It seems to happen during our windows updates, and I'm guessing all the reboots and such confuse the hell out of it.

                                             

                                            Is there a way to put IPAM in a maintenance mode to pause scanning during updates.

                                             

                                            Wondering if the Solarwinds Servers should be set to manually reboot from updates?  Then once all other servers have been updated and rebooted. Then reboot Solarwinds servers.

                                             

                                            Curious if that is the same situation others are seeing.

                                          • Re: IPAM Scan job Status question
                                            darcydressell44

                                            Majorly having this issue.

                                            • Re: IPAM Scan job Status question
                                              lauminshen

                                              I do hope SW address this issue. During my last upgrade (4.5 -> 4.7), I hit a lot of issue with the scan jobs...