13 Replies Latest reply: Jul 19, 2012 10:14 AM by cweeklund RSS

Stagger SeUM transaction on a player?

netlogix

Is there a way to stagger the transaction on a player?  My cloud player server pegs itself out for 30-45 seconds every five minutes.  I am wondering if that is one of the causes for some fluctuation in the response times reported.  If I could tell the player to only work on one transaction at a time (serial vs parallel), I think it would spread that utilization out and get better response times.  Sure once I get to 50 transaction being handled by one player, it wouldn't keep up, but shouldn't I add another player if that happened?

 

Is there a way to do this?

 
  • Re: Stagger SeUM transaction on a player?
    aLTeReGo

    While it's certainly possible to change the number of parallel threads the SeUM Player uses, from the default 2 for a SeUM Player installed on the Orion server or 7 for standalone Player installations with the assistance of support, the better option would be to add additional SeUM Players to your environment or reduce the playback interval for your transactions currently assigned. Simply reducing the number of parallel threads the SeUM Player uses is likely to result in more issues then it solves causing transactions not being played back on time, or in some cases at all.

    • Re: Stagger SeUM transaction on a player?
      netlogix

      I only have 5 Monitors on the node now, so the load is already light. I just want the load spread out.  The average duration of all the transactions is about 2 minutes and the rate is once every 5.  So that should be 40% of the time on one thread.  Might be better to do 2 threads to keep it down closer to 20%.  I'll work with support.

      • Re: Stagger SeUM transaction on a player?
        aLTeReGo

        In the scenario you describe above reducing the number of Threads is likely to cause transactions to be delayed in their playback. This may reduce resource overhead because fewer jobs are able to run, but this is because you'll have created an artificial bottleneck in the number of simultaneous transactions that can be executed. Essentially the same result can be achieved by simply increasing the playback interval for these transactions to once every 7 or 10 minutes, as that's essentially what will be occurring anyway once you bottleneck the thread count. The SeUM Player may come pre-configured for seven parallel threads but it will never use more than it needs to achieve the monitoring requirements as configured when the transaction is assigned to the player. For example, as the SeUM Player comes configured by default it will use only one thread to keep monitoring as many transactions as that thread can handle until another scheduled job (as scheduled by the configured playback frequency interval) overlaps with a currently executing thread. Only when this occurs does another thread begin to execute. It sounds like you're already at this point where the number of active threads is exceeding your desired resource overhead. Bottle necking the number of simultaneous transactions that can be executed will only delay the scheduled execution of the configured playback interval, and in a worst case scenario you may find that some transactions are never being played back because there's simply never any available threads for the transaction to be played back. The best approach would be to stretch out the playback interval so there's never more than the desired number of threads active at any given time. You can monitor this by watching the number of SolarWinds.SEUM.Agent.Worker.exe processes running through Task Manager.

        • Re: Stagger SeUM transaction on a player?
          netlogix

          I figured out how to make the change and it works way better... can you guess when I enabled it based on these screen shots?

          1.jpg

           

          Also, everything is completing faster now.  The total time for the 5 transactions is around 100 sec, which mean the player is busy for only 33% of the time.  I don't understand what the down side would be.  I am also getting far fewer "Critical" or "Warnings".  I am not running very many transaction; five each with an average duration of 20 seconds.  I should be able to complete them easily within the 300 second interval.  I don't know if I was clear last time that the total of the 5 was 2 minutes not the duration of each.

          • Re: Stagger SeUM transaction on a player?
            taylorfc

            So did you change your polling interval to 10 minutes?

            • Re: Stagger SeUM transaction on a player?
              netlogix

              I changed the \SolarWinds\Orion\SEUM\Player\SolarWinds.SEUM.Agent.Service.exe.config agentConfiguration section to contain numWorkerProcesses="1" maxPlaybacksPerWorker="1".  But mind what alterego said, if your agent doesn't have enough time to complete all polls, you could cause missed polls with this configuration.

              • Re: Stagger SeUM transaction on a player?
                taylorfc

                I don't like the idea of a sequential playback instead of parallel since it won't scale.  I do see that many of your transactions talk a long time to execute(30 + seconds), and not knowing the actual amount of rendering needed for each transaction I can't say I disagree with the sequential approach.  I've had to code a hybrid approach in the past.  Needed 250 web site monitors to execute in a monitoring program I wrote some years back.  I pooled 50 threads(configurable) as workers to download the web pages.  I has the exact same issue, the server would get hammered the first 60 seconds or so.  This had a negative impact on the quality of the response times(they would be inconsistent).  My fix was to put in a configurable buffer between each monitor execution.  The buffer could either be a hard number, ie 3000ms, or a random value between a range(50ms to 5000ms).  Pro's and Con's to this approach.

                 

                PRO's: The response times flattened and the server was much more responsive during polling.  Due to the buffer between the monitor execution in each thread, the system resource usage was eased.  The randomized sleep period worked best, as it worked will with the random nature of the monitor execution response times.  It also allowed the threads to "spin up" instead of slamming the petal to the metal.

                 

                CON's:  Total time needed to execute all monitors was longer.  Mind you that it was not much longer.

                 

                Don't know if SolarWinds would consider this approach.  I built multiple monitoring engines(snmp/http/icmp) from the design and they all worked well and scaled well.  The code change would be small and the customers would benefit.

          • Re: Stagger SeUM transaction on a player?
            cweeklund

            I changed the config file to contain those max values of one and my stagger is gone, but the first step of each transaction now takes about 7 seconds.  Any thoughts on what we can do here?  This is a dedicated VM with only 7 monitors, none of them doing much more than login, enter value, check response, every 5 minutes.

            • Re: Stagger SeUM transaction on a player?
              jiri.tomek

              SEUM clears browser cache before each playback so that may cause higher duration of first step because all content must be downloaded again.

              • Re: Stagger SeUM transaction on a player?
                cweeklund

                We have an external player at Amazon, which does not have the staggered playback issue.  The stagger is on our internal player, which is subject to various GPO, antivirus, etc.  We have a user GPO to empty temporary internet files on IE exit that applies to my domain user account, but it's my understanding that the player runs under the local SEUM-User-# accounts, which are not domain users.  I have a ticket open with support, and their recommendation was to disable the policy that was forcing the browser cache to empty each time - which had no effect, presumably since the user account is not a domain user.