7 Replies Latest reply on Jun 13, 2016 1:04 PM by milewideinchdeep

    Cannot collect data on additional pollers, [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!

    kenshin0402

      Hi

      My additional pollers cannot collect data, or I should say the data is discontinuous. But my primary poller works well.

       

      Like this:

      t.JPG

       

      I find logs on my primary poller servers:

       

      *** SolarWinds Job Engine v1.6.0.28, .Net Runtime v4.0.30319 ***
      2014-03-08 10:45:11,280 [WorkerProcessMonitor] INFO  SolarWinds.JobEngine.Engine.WorkerMonitor - Job monitoring starting
      2014-03-08 10:45:11,296 [6] INFO  SolarWinds.JobEngine.PluginStore - Refreshing using C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins as plug-ins manifest container
      2014-03-08 10:45:11,311 [6] INFO  SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.Core.Discovery.Jobs.plugin"
      2014-03-08 10:45:11,327 [6] INFO  SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.Core.Jobs.plugin"
      2014-03-08 10:45:11,327 [6] INFO  SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.IpSla.Jobs.plugin"
      2014-03-08 10:45:11,327 [6] INFO  SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.IpSla.Jobs.SNMP.plugin"
      2014-03-08 10:45:16,475 [6] INFO  SolarWinds.JobEngine.ServiceProcess.SolarWindsService - OnStart: Application started
      2014-03-08 10:46:19,459 [QueueMonitoringManager] INFO  SolarWinds.JobEngine.Engine.WorkerProcessWCFProxy - worker process 78b4a84d-5ad7-47ac-aae1-c99d14e33087 has pid 11140
      2014-03-08 10:50:15,342 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-08 23:10:17,601 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-08 23:15:17,676 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 03:40:18,739 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 03:45:18,740 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 03:50:18,878 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 03:55:18,941 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 14:30:20,295 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 15:50:20,766 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:20:24,058 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:25:24,576 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:30:25,512 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:35:25,851 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:40:25,992 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:45:26,059 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:50:26,087 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 17:55:26,108 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:00:26,130 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:05:26,150 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:10:26,171 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:15:26,192 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:20:26,213 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:25:26,234 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:30:26,255 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:35:26,276 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:40:26,297 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:45:26,318 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:50:26,339 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 18:55:26,360 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:00:26,382 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:05:26,402 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:15:27,834 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:20:27,855 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:25:27,876 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:30:27,897 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:35:27,824 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:40:27,756 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:45:27,892 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:50:27,961 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 19:55:28,000 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:00:28,021 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:05:28,041 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:10:28,062 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:15:28,083 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:20:28,104 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:25:28,125 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:30:28,146 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:35:28,167 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:40:28,188 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:45:28,209 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:50:28,230 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 20:55:28,251 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:00:28,272 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:05:28,294 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:10:28,315 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:15:28,336 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:20:28,357 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:25:28,378 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:30:28,399 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:35:28,420 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:40:28,441 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
      2014-03-09 21:45:28,462 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!

       

       

      What is 'Woke up too early' ?!

       

      Other logs on my primary poller:

       

      2014-03-08 10:45:18,191 [6] INFO  SolarWinds.JobEngine.Scheduling.SchedulerServiceApp - Database versions match: 2

      2014-03-08 10:45:18,284 [6] INFO  SolarWinds.JobEngine.Scheduling.CredentialHandlerFactory - Using certificate for credentials protection

      2014-03-08 10:45:18,362 [6] INFO  SolarWinds.JobEngine.Broker.JobExecutionEngineManager - Wait period between retry attempts will be: 300 seconds

      2014-03-08 10:45:18,394 [6] INFO  SolarWinds.JobEngine.Broker.ActiveJobJournal - Initialized with 0 active jobs

      2014-03-08 10:45:22,621 [6] INFO  SolarWinds.JobEngine.Broker.ActiveJobJournal - Retrieved 0 incomplete jobs

      2014-03-08 10:45:23,838 [6] INFO  SolarWinds.JobEngine.ServiceProcess.SolarWindsService - OnStart: Application started

      2014-03-08 10:46:12,699 [14] INFO  SolarWinds.JobEngine.Broker.JobEngineWCFProxy - job engine 86547d36-4cce-43e4-86e6-1bf994a47e54 located at net.pipe://poller001/solarwinds/jobengine/engine

      2014-03-08 10:46:25,045 [ResultDispatcher] WARN  SolarWinds.JobEngine.Scheduling.JobScheduler - Notification uri not found [net.pipe://localhost/orion/ipsla/servicemonitor/echo]. Next 100 endpoint not found will be suppressed

      2014-03-08 10:46:26,061 [ResultDispatcher] WARN  SolarWinds.JobEngine.Scheduling.JobScheduler - Notification uri not found [net.pipe://localhost/orion/ipsla/servicemonitor/echo]. Next 100 endpoint not found will be suppressed

      2014-03-08 10:47:04,604 [5] INFO  SolarWinds.JobEngine.Broker.JobEngineWCFProxy - job engine f7c98c62-d254-4d09-bee4-287d9033f084 located at net.tcp://poller002:17777/solarwinds/jobengine/engine

      2014-03-08 10:47:05,624 [5] INFO  SolarWinds.JobEngine.Broker.JobEngineWCFProxy - job engine f7c98c62-d254-4d09-bee4-287d9033f084 located at net.tcp://poller002:17777/solarwinds/jobengine/engine

      2014-03-08 10:47:06,663 [5] WARN  SolarWinds.JobEngine.Broker.JobEngineInfo - Error reseting proxy

      SolarWinds.JobEngine.Broker.JobEngineNotListeningException: The job engine f7c98c62-d254-4d09-bee4-287d9033f084 is not listening ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://poller002:17777/solarwinds/jobengine/engine/ssl. The connection attempt lasted for a time span of 00:00:01. TCP error code 10061: No connection could be made because the target machine actively refused it 10.x.x.3:17777.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.x.x.3:17777

         at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

         at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)

         at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)

         --- End of inner exception stack trace ---

       

      Server stack trace:

         at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)

         at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)

         at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)

         at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedFramingRequest.SendRequest(Message message, TimeSpan timeout)

         at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)

         at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

         at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)

         at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)

         at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)

         at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)

         at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

         at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)

         at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

         at System.ServiceModel.Channels.CommunicationObject.Open()

       

      Exception rethrown at [0]:

         at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

         at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

         at System.ServiceModel.ICommunicationObject.Open()

         at SolarWinds.JobEngine.Broker.JobEngineWCFProxy.EstablishConnection(MessageCredentialType credentialType)

         at SolarWinds.JobEngine.Broker.JobEngineWCFProxy.EstablishConnection(MessageCredentialType credentialType)

         at SolarWinds.JobEngine.Broker.JobEngineWCFProxy.Connect()

         at SolarWinds.JobEngine.Broker.JobEngine.Reset()

         --- End of inner exception stack trace ---

         at SolarWinds.JobEngine.Broker.JobEngine.Reset()

         at SolarWinds.JobEngine.Broker.JobEngineInfo.Reconnect()

       

       

      Is there anybody know how to solve it?

       

      Thanks in advance.

       

      /Alex