18 Replies Latest reply on Jan 4, 2011 5:50 AM by Questionario

Connection timeout. Job cancelled by scheduler

aLTeReGo Apr 16, 2010 10:30 AM

I've been living my own little personal APM hell since upgrading my Orion to RC v10 on Windows Server 2008 R2 SP2. I'm running the latest version of APM 3.5 RC2, as well as RC2 for Orion v10 but my APM issues continue. I'm having a very difficult time with one template applied to a specific server in particular where I'm receiving the error "Connection timeout. Job cancelled by scheduler". I've also determined that new component monitors that are added to otherwise working templates are also failing. They mostly say "Initial Poll in Progress" forever. I have a lengthy case open (Case #152518) that outlines the troubles I've been encountering, as well as includes a few SolarWinds Diagnostics.

If there's anyone on the Dev team who might be able to offer up some insight as to what might be causing these erroneous errors I'd love to hear from you. In case there was any doubt, yes these templates and monitors were all once working perfectly under APM 3.1 on my Windows Server 2003 Orion v9.5.1 machine. Why they don't work now is anyone's guess.

Re: Connection timeout. Job cancelled by scheduler

Peter.Cooper Apr 16, 2010 2:40 PM (in response to aLTeReGo)

Hi Jeremy,
I'm working your case with Oliver. I think we're trying to nail down either a) why does the application take a hair over 20 minutes to complete, b) if this is status quo, why is it complaining after upgrade.
One thing I'm looking into is if we've previously modified solarwinds.apm.probes.dll.config (it has a timeout multiplier setting)... and if so, was the file reverted back to no longer having those modifications.
Peter
Like (0)

Actions
- Re: Connection timeout. Job cancelled by scheduler
  
  aLTeReGo Apr 16, 2010 3:16 PM (in response to Peter.Cooper)
  
  Thanks Peter for assisting Oliver. I've attached my solarwinds.apm.probes.dll.config file to the case but I think you'll find that it's very ordinary. I've made no modifications to the file and this is a new install.
  I'm more than a little perplexed how a response made to a request made 20 minutes prior could even happen. Shouldn't this timeout somewhere else in the chain long before we hit 20 minutes? Or is this how long it sits in the job queue before it's processed by the job scheduler?
  
  Like (0)
  
  Actions
Re: Connection timeout. Job cancelled by scheduler

chris.lapoint Apr 16, 2010 2:57 PM (in response to aLTeReGo)

Dev is investigating now. As you said, it's strange that we haven't seen this from other customers so it may be environmental. In any case, we're absolutely committed to getting to the bottom of this issue
Like (0)

Actions
- Re: Connection timeout. Job cancelled by scheduler
  
  aLTeReGo Apr 16, 2010 3:19 PM (in response to chris.lapoint)
  
  Dev is investigating now. As you said, it's strange that we haven't seen this from other customers so it may be environmental. In any case, we're absolutely committed to getting to the bottom of this issue
  Chris, I have absolutely no doubt whatsoever that your team will get to the bottom of what's going on. Even if it's something stupid and totally my fault ;-)
  
  Like (0)
  
  Actions
  - Re: Connection timeout. Job cancelled by scheduler
    
    bwicks Apr 17, 2010 10:49 PM (in response to aLTeReGo)
    
    Not sure if related but I am also getting Connection timeout. Job canceled by scheduler. in NPM 9.5.1 and APM 3.1 - Our case is 152579
    We verified with a packet capture on our Orion server that our APM - SNMP component monitor was not attempting any SNMP polls when getting the "connection timeout" error. Causing the status to be unknown.
    
    Thanks
    
    Like (0)
    
    Actions
    - Re: Connection timeout. Job cancelled by scheduler
      
      MarieB Apr 19, 2010 3:10 PM (in response to bwicks)
      
      Hi Jeremy--
      When this gets resolved, would you post the results on thwack?
      Many thanks,
      M
      
      Like (0)
      
      Actions
      - Re: Connection timeout. Job cancelled by scheduler
        
        Peter.Cooper Apr 20, 2010 8:40 AM (in response to MarieB)
        
        Update on the original case: MAPI API appears to be hanging on Win2k8 R2 without the CDO 1.2.1 update. The MSDN blog says that applying this update is mandatory on Win2k8, but this is the first time I've seen it hang due to the drivers not being updated.
        Peter.
        
        Like (0)
        
        Actions
      - Re: Connection timeout. Job cancelled by scheduler
        
        aLTeReGo Apr 22, 2010 9:57 AM (in response to MarieB)
        
        I had a couple of issues. The first being that I needed to install the Messaging API and Collaboration Data Objects 1.2.1 for the MAPI user experience monitor to work with Windows Server 2008 R2. Then we discovered that the MAPI Profile was not being created properly. We had to use the MFCMAPI program to repair the profile.
        My last issue was a Performance Counter Monitor that was found and added via the "Find Processes, Services, and Performance Counters" wizard which did not actually exist using Perfmon. Removing that fictitious monitor allowed the rest of the component monitors in the template to function normally.
        
        Like (0)
        
        Actions
        
        Re: Connection timeout. Job cancelled by scheduler
        
        aLTeReGo Apr 22, 2010 10:15 AM (in response to aLTeReGo)
        
        To expand on the APM profile not being created properly. APM creates the MAPI profile but does not associate an Exchange Mailbox to the profile leaving the monitor broken with the error "Unexpected error code returned by SolarWinds.APM.MAPI.exe: -1073741819"
        Using the MFCMAPI tool you simply right click on the MAPI profile created by APM and click "Add Exchange Mailbox". Then you type in the server name or IP address and the exchange mailbox name (usually the username). After that the monitor should test properly.
        
        Like (0)
        
        Actions
        
        Re: Connection timeout. Job cancelled by scheduler
        
        Peter.Cooper Apr 23, 2010 8:32 AM (in response to aLTeReGo)
        
        aLTeReGo, thanks for keeping us informed in real time. I appreciated your contact yesterday morning. We've got a replication going in house now. We've got the hang replicated and moving toward replicating the rest of your experience.
        Edit: Okay, we have the whole thing replicated locally!
        Peter.
        
        Like (0)
        
        Actions
        
        Re: Connection timeout. Job cancelled by scheduler
        
        justin0469 May 12, 2010 9:04 AM (in response to Peter.Cooper)
        
        any updates on this?
        
        Like (0)
        
        Actions
        
        Re: Connection timeout. Job cancelled by scheduler
        
        Questionario May 12, 2010 9:15 AM (in response to Peter.Cooper)
        
        is this fixed in the final release now?
        Because thats one of the major things we bought APM for...
        Does this work in APM 3.1?
        
        Like (0)
        
        Actions
        
        Re: Connection timeout. Job cancelled by scheduler
        
        byrona May 12, 2010 10:01 AM (in response to Peter.Cooper)
        
        We have been experiencing these problems with APM using WMI probes. I have an open support case on it (#156527) where the Tech has suggested that I need to re-install all of my Orion software. He believes the cause is the fact we have installed some patches and updates not using the Local Admin account (we used a domain account with admin access) which has caused permission issues with some DLL files.
        Needless to say, this is a very frustrating issue.
        
        Like (0)
        
        Actions
        
        Re: Connection timeout. Job cancelled by scheduler
        
        mark wiggans May 12, 2010 10:07 AM (in response to byrona)
        
        Yes- Fixed memory leaks in MAPI User Experience and Windows Script component monitors in version 3.5 which is out now...
        
        http://www.solarwinds.com/support/apm/docs/ReleaseNotes/releaseNotes.htm
        
        Like (0)
        
        Actions
    - Re: Connection timeout. Job cancelled by scheduler
      
      Peter.Cooper Apr 20, 2010 8:46 AM (in response to bwicks)
      
      bwicks - I had a peek at your case. I'd like to know when your reinstallation is finished and we're back onto working with your support rep to figure out the issue. Cheers!
      Peter.
      
      Like (0)
      
      Actions
Re: Connection timeout. Job cancelled by scheduler

firehawk_350 Aug 3, 2010 10:20 AM (in response to aLTeReGo)

We are having the same issue with APM 3.5 RC2 and working with Oliver as well.. Sounds like the same setup(Win2K) our ticket # is 175823
Like (0)

Actions
- Re: Connection timeout. Job cancelled by scheduler
  
  MrSampsonite Dec 22, 2010 11:30 PM (in response to firehawk_350)
  
  did you ever see a resolution to this? we are on 3.5 and just had all our disk stats go to unknown for the last couple of hours. very odd.
  
  Like (0)
  
  Actions
  - Re: Connection timeout. Job cancelled by scheduler
    
    Questionario Jan 4, 2011 5:50 AM (in response to MrSampsonite)
    
    I guess some of the stuff got resolved...
    we have a ton more going unknown, we sometimes have all WMI monitors on a server go unknown and stay this way for hours or days. luckily in the next version there will be an alternative for WMI :)
    
    Like (0)
    
    Actions

Go to original post