4 Replies Latest reply on Mar 5, 2013 12:56 AM by jiri.tomek

    player ran out of disk space and now transactions won't run

    romeyde

      Sometime over the weekend one of my players ran out of diskspace due to debug being turned on and having several weeks of log files on it. 

      I cleaned up disk space but still haven't been able to get transactions to run.  They stay in an 'Unknown' state.

      I've tried restarting the service and even rebooted the player server.

       

      After a short time, I'm seeing the primary process climb to 2GB of memory and the CPU usage in the web gui is currently being reported at 501%.

      I see the individual user processes kick in but immediately exit.

       

      I am guessing I can "fix" this by deleting the player and re-adding it but this isn't ideal.   If we were in PRD, this would cause serious gaps in monitoring and we would lose all history by doing this and having to re-add transactions to the player.

       

      (this is a good example as to why I requested a "pool of players" as a feature request, this way, a bad player doesn't completely mess up monitoring for 50+ transactions, just take the bad player out of the pool, re-install it and put it back in the pool)

       

      I did open a support ticket and waiting a response.  Trying to wait before I go ahead and just delete the player and re-install since again, we don't want this happening in production.

       

       

       

       

      I'm seeing this in the AgentService log

       

      2013-03-04 12:09:12,159 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Disposing worker process launcher 4792603a-f51f-4c1d-a67f-0ff649a29e55

      2013-03-04 12:09:12,159 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - Unable to create worker process. Trying again.

      SolarWinds.SEUM.Agent.Exceptions.AgentException: Unable to launch worker process 4792603a-f51f-4c1d-a67f-0ff649a29e55

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy.Connect(Boolean launchProcess)

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy..ctor(String userName, String domain, String password, Int32 sessionId)

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessFactory.CreateWorkerProcess(String username, String domainOrMachine, String password, Int32 sessionId)

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager.AddWorkerProcess(String username, String domainOrMachine, String password, Boolean runInInteractiveSession)

      2013-03-04 12:09:12,159 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - Unable to create worker process after 3 retries.

      2013-03-04 12:09:12,159 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - } AddWorkerProcess exited

      2013-03-04 12:09:12,159 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - No available user account to start new worker process.

      2013-03-04 12:09:12,159 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - } FindWorkerProcess exited

       

      And this in the AgentService.log

       

      2013-03-04 12:08:45,358 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Disposing worker process launcher 3d9b616d-721c-4fb6-b2b1-32957c7689aa

      2013-03-04 12:08:45,358 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - Unable to create worker process. Trying again.

      SolarWinds.SEUM.Agent.Exceptions.AgentException: Unable to launch worker process 3d9b616d-721c-4fb6-b2b1-32957c7689aa

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy.Connect(Boolean launchProcess)

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy..ctor(String userName, String domain, String password, Int32 sessionId)

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessFactory.CreateWorkerProcess(String username, String domainOrMachine, String password, Int32 sessionId)

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager.AddWorkerProcess(String username, String domainOrMachine, String password, Boolean runInInteractiveSession)

      2013-03-04 12:08:45,358 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy - Running WorkerProcess registration service at: net.pipe://localhost/seum/agent/33d27d0a-afd5-4999-a289-15c6de46ae1c

      2013-03-04 12:08:45,374 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Launching Worker Process 33d27d0a-afd5-4999-a289-15c6de46ae1c as user SEUM-User-3

      2013-03-04 12:08:45,389 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Error launching worker process.

      System.Exception: Error launching worker process. Error: process creation error. System error: 0x3ee

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher.LaunchWorkerProcessInCurrentSession()

         at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher.LaunchWorkerProcessInternal()

       

       

      Orion Core 2012.2.1, WPM 2.0.1



        • Re: player ran out of disk space and now transactions won't run
          jiri.tomek

          Hello,

          try to stop player service, go to Computer Management -> Local Users and Groups and delete all "SEUM-User-X" accounts. Then restart the service. Check that AgentService.log file does not contain those errors anymore. This could help.

            • Re: player ran out of disk space and now transactions won't run
              romeyde

              Stopped service, removed users and restarted service.   After restarting service I saw it make all of these new "SEUM-User-#.<NODENAME>" directories under C:\Users.  So now I have directories with and without <NODENAME> there.

              But, I am still seeing this in the log:

               

              2013-03-04 13:07:38,131 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Disposing worker process launcher 5628cd8e-2808-4cd2-bb43-2c7ba2cf02d9

              2013-03-04 13:07:38,131 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - Unable to create worker process. Trying again.

              SolarWinds.SEUM.Agent.Exceptions.AgentException: Unable to launch worker process 5628cd8e-2808-4cd2-bb43-2c7ba2cf02d9

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy.Connect(Boolean launchProcess)

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy..ctor(String userName, String domain, String password, Int32 sessionId)

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessFactory.CreateWorkerProcess(String username, String domainOrMachine, String password, Int32 sessionId)

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager.AddWorkerProcess(String username, String domainOrMachine, String password, Boolean runInInteractiveSession)

              2013-03-04 13:07:38,131 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy - Running WorkerProcess registration service at: net.pipe://localhost/seum/agent/14e8dcf3-db93-4d9a-8d94-011cc0bf3c8c

              2013-03-04 13:07:38,131 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Launching Worker Process 14e8dcf3-db93-4d9a-8d94-011cc0bf3c8c as user SEUM-User-7

              2013-03-04 13:07:38,131 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Error launching worker process.

              System.Exception: Error launching worker process. Error: process creation error. System error: 0x3ee

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher.LaunchWorkerProcessInCurrentSession()

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher.LaunchWorkerProcessInternal()

              2013-03-04 13:07:38,131 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessLauncher - Disposing worker process launcher 14e8dcf3-db93-4d9a-8d94-011cc0bf3c8c

              2013-03-04 13:07:38,131 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - Unable to create worker process. Trying again.

              SolarWinds.SEUM.Agent.Exceptions.AgentException: Unable to launch worker process 14e8dcf3-db93-4d9a-8d94-011cc0bf3c8c

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy.Connect(Boolean launchProcess)

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessWCFProxy..ctor(String userName, String domain, String password, Int32 sessionId)

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessFactory.CreateWorkerProcess(String username, String domainOrMachine, String password, Int32 sessionId)

                 at SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager.AddWorkerProcess(String username, String domainOrMachine, String password, Boolean runInInteractiveSession)

              2013-03-04 13:07:38,131 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - Unable to create worker process after 3 retries.

              2013-03-04 13:07:38,131 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - } AddWorkerProcess exited

              2013-03-04 13:07:38,131 [PlaybackDispatcher] ERROR SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - No available user account to start new worker process.

              2013-03-04 13:07:38,131 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - } FindWorkerProcess exited

              2013-03-04 13:07:39,145 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - { FindWorkerProcess entered

              2013-03-04 13:07:39,145 [PlaybackDispatcher] DEBUG SolarWinds.SEUM.Agent.WorkerProcessManagement.WorkerProcessManager - { OnBeforeFindWorkerProcess entered