8 Replies Latest reply on Oct 17, 2012 11:50 AM by byrona

    Events Per Second?

    byrona

      What is the EPS capacity of a LEM appliance?  I have dug though the documentation but have not been able to find this information.

        • Re: Events Per Second?
          byrona

          I really need to get this information, is this something I would need to call support to get?

            • Re: Events Per Second?
              Jeffster

              HI Byrona,

               

              I'll bet this is a question that makes the support techs cringe and want to runaway.......

               

              Is yours a virtual appliance?  If so, then it is really going to depend on the resources/reservations you have made at the host level for your appliance.  We are using HyperV on 2008r2 on a Dell 710R with dual xeon x5570 and right now 12GB of ram (soon to be 64GB so we can allocate 32GB to the appliance).  We have the virtual machine set to use upto 16 logical processors with 25% reserved for the LEM and the weight set to 100%.    Now I picked a random 1week interval and came up just shy of 128 million (127,947,883) results.  Divided by 7 to get to days, divided by 24 to get to hours, devided by 60 to get to minutes and then again by 60 to get my eps average for the week and came up with 211 eps.  I looked for the busy day and started narrowing down until I found the busiest section and came up with a 10s Window that had 19155 for a 10s average of 1915eps.  Within that window I found a 2s window with 5129 for a 2564eps average.  The time actually broke down to 2531 in the 1st second and 3198 in the 2nd second.

               

              So, based on my configuration I have seen as high as 3200 events per second.  The insertion time vs detection time was usually a zero to one second variant but I found it to be as high as 5 seconds but I don't know if that is normal or not.  If I could figure out a way to reliably search by insertion time it would be better events per second collection rate numbers.

               

              We are monitoring 25 servers and 13 network devices and we log everything.  And everything means everything.  That is why we are increasing the RAM is to hopefully keep up with a particular scenario where events can be dropped/lost which is unacceptable.

                • Re: Events Per Second?
                  byrona

                  Our system is a VMWare virtual appliance and has the out-of-box configuration with 8GB of RAM and 2 vCPU's.  We aren't running near any potential capacity issues now.  I am looking for the EPS at both a burst capacity and a sustained capacity for architectural decisions going forward.  As a service provider we are building security and event management services based on LEM and potentially Tripwire so I need to know what the system is capable of.

                   

                  Thanks for sharing this information.

                  • Re: Events Per Second?
                    Jeffster

                    A few more details, I found that our processing limitation seems to be around 670 eps.  We had a system issue that caused events to back up on the windows servers and flood into the appliance.  According to my web console, the system sat around 40000 events per minute for about 45minutes.  Simple math shows that we max out processing around 670 events per second.  Now, the system will actually queue the files up on both the agent and the appliance (up to a point and then drop) and I've seen my waiting queue up around 1.3 million events waiting to be processed which seems to be the upper limit of our resource configuration.  Still waiting to here from support on our issue.    Otherwise it works.

                • Re: Events Per Second?
                  byrona

                  I called SolarWinds support and asked them about this.  According to them it really is dependent on the amount of resources you give it.  So long as you keep adding more dedicated CPU and Memory it should be able to keep taking in more and more events.  The only resource limitation is on the disk, the maximum disk space the current v5.4 is 2 TB.

                   

                  To also help increase the speed and decrease the processing time the appliance takes you can add a Linux syslog server with an agent that all of your "chatty" networking gear logs to so all of the normalization takes place on the Linux system and then that data is forwarded on to the LEM appliance.

                  2 of 2 people found this helpful