Billions of Jobs Queued???

Why are jobs queuing up in the billions? see attached. I also verified this using performance monitor on the server itself. Polling percentage was 98% but >2 billion jobs queued.

Job Engine v2 Jobs Queue .xlsx
  • It sounds like jobs are queued on the poller itself, or there are orphaned jobs. Restarting SW services would likely take care of the immediate issue. If it reoccurs, a support ticket would probably be the best course of action to get to the root cause.

  • Jobs Queue is performance counter with type INT32. In case counter is decremented and original value is zero, it goes into negative value. But because it's signed int and it can't be negative that's why this value rollover and counter has its maximum value 2,147,483,647.

    Explanation: someone restarted all counters with lodctr.exe, or there could be some synchr. issue inside jobengine with increment/decrement counter value.

    Anyway this is harmless behavior and it has no impact on your system. As Rob already mentioned, restart service will fix it.

  • Ok. So I restarted the job engine V2 service and that cleared out the counters and the issue. The issue was that, none of the Application monitors were polling. Which is what prompted me to check the performance counters and I found the Job Engine V2 Jobs queue way out of wack.

    I'll open up a case and get to the bottom of this. Thanks for the help. I'll post a root cause if or when we find one.

  • In my case it was not harmless behavior because the Applications monitors stopped working. So the evidence did not point to a counter error as you can see in the attached file of the original post.