Before I go an open a ticket with support, I'm curious if anybody else has had issues with the SAM Agent no longer sending data to SAM? Basically the behavior we're seeing is that after an amount of time (seems to be about 40 days on 4 of the boxes I've seen the issue on), the agent stops sending data back to SAM. By this, I mean all the components being monitored go to "unknown", and we get an error for some items on our UI. The annoying thing is it says the agent is still running, but clearly it's not running properly.
Notice the little grey square?
Usually a quick restart of the agent, and everything is fine again. I'm wondering if I'm hitting a counter limit or some other kind of limit that pops up after a certain amount of time.
One interesting log file I found was C:\ProgramData\SolarWinds\JobEngine.v2\Logs_Agent\SolarWinds.JobEngineService_v2_10_agent.log, which seemed to be complaining about not knowing the origin, and a file already existing (but not what file).
2016-03-11 12:23:11,399 [STP SmartThreadPool Thread #0] INFO SolarWinds.JobEngine.Engine.WorkerControllerWCFProxy - Started new worker process with pid 107402016-03-11 12:23:16,243 [STP SmartThreadPool Thread #0] INFO SolarWinds.JobEngine.Engine.SharedWorkerProcessWCFProxy - worker fc3e69ca-ebff-433f-8776-3320caac7440 started in worker process with pid 107402016-03-11 12:23:16,274 [STP SmartThreadPool Thread #0] INFO SolarWinds.JobEngine.Engine.WorkerProcessManager - Added worker process for 'SolarWinds.APM.Probes' 64bit=True: fc3e69ca-ebff-433f-8776-3320caac74402016-03-11 12:23:35,961 [ResultWorker0] ERROR SolarWinds.JobEngine.AgentSupport.JobResultsProcessor - Agent job results processor does not know origin of job 'daf6c44a-f33d-4f47-bccc-6d37b82bdbd5'. Unable to process job results.2016-03-11 12:24:23,358 [7] ERROR SolarWinds.JobEngine.AgentSupport.MessageHandler - Error processing incoming message: ID: 68147ece-7e3b-4fc9-a3ab-3e8dc419453b Source: 3021c5db-e99c-4d2d-b3aa-5a18a26eb6b7 Sender: JobEngine Target: cf616ad8-c81b-473b-b4e9-adb5bde3697a Recipient: JobEngine Type: GetJobResultStream Timeout: 00:02:00 RemainingTimeout: 00:01:55.8098723 CreationTime: 3/11/2016 6:24:19 PM, Files: 0 ResponseStatus: ValidResponse ResponseTo: 00000000-0000-0000-0000-000000000000 Data: <guid xmlns="http://schemas.microsoft.com/2003/10/Serialization/">87583500-08a9-4ad4-adc4-468d58283a86</guid>2016-03-11 12:24:23,358 [7] ERROR SolarWinds.JobEngine.AgentSupport.MessageHandler - Error:System.IO.IOException: The file exists. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.__Error.WinIOError() at System.IO.Path.InternalGetTempFileName(Boolean checkHost) at SolarWinds.JobEngine.AgentSupport.MessageHandler.GetJobResultStream(Message message) at SolarWinds.JobEngine.AgentSupport.MessageHandler.MessageReceived(Object sender, Message message)
All the boxes are running agent 1.2.0.78, and SAM is 6.2.1.
