Events Per Second?

I really need to get this information, is this something I would need to call support to get?

HI Byrona,

I'll bet this is a question that makes the support techs cringe and want to runaway.......

Is yours a virtual appliance?  If so, then it is really going to depend on the resources/reservations you have made at the host level for your appliance.  We are using HyperV on 2008r2 on a Dell 710R with dual xeon x5570 and right now 12GB of ram (soon to be 64GB so we can allocate 32GB to the appliance).  We have the virtual machine set to use upto 16 logical processors with 25% reserved for the LEM and the weight set to 100%.    Now I picked a random 1week interval and came up just shy of 128 million (127,947,883) results.  Divided by 7 to get to days, divided by 24 to get to hours, devided by 60 to get to minutes and then again by 60 to get my eps average for the week and came up with 211 eps.  I looked for the busy day and started narrowing down until I found the busiest section and came up with a 10s Window that had 19155 for a 10s average of 1915eps.  Within that window I found a 2s window with 5129 for a 2564eps average.  The time actually broke down to 2531 in the 1st second and 3198 in the 2nd second.

So, based on my configuration I have seen as high as 3200 events per second.  The insertion time vs detection time was usually a zero to one second variant but I found it to be as high as 5 seconds but I don't know if that is normal or not.  If I could figure out a way to reliably search by insertion time it would be better events per second collection rate numbers.

We are monitoring 25 servers and 13 network devices and we log everything.  And everything means everything.  That is why we are increasing the RAM is to hopefully keep up with a particular scenario where events can be dropped/lost which is unacceptable.

Our system is a VMWare virtual appliance and has the out-of-box configuration with 8GB of RAM and 2 vCPU's.  We aren't running near any potential capacity issues now.  I am looking for the EPS at both a burst capacity and a sustained capacity for architectural decisions going forward.  As a service provider we are building security and event management services based on LEM and potentially Tripwire so I need to know what the system is capable of.

Thanks for sharing this information.

I called SolarWinds support and asked them about this.  According to them it really is dependent on the amount of resources you give it.  So long as you keep adding more dedicated CPU and Memory it should be able to keep taking in more and more events.  The only resource limitation is on the disk, the maximum disk space the current v5.4 is 2 TB.

To also help increase the speed and decrease the processing time the appliance takes you can add a Linux syslog server with an agent that all of your "chatty" networking gear logs to so all of the normalization takes place on the Linux system and then that data is forwarded on to the LEM appliance.

A few more details, I found that our processing limitation seems to be around 670 eps.  We had a system issue that caused events to back up on the windows servers and flood into the appliance.  According to my web console, the system sat around 40000 events per minute for about 45minutes.  Simple math shows that we max out processing around 670 events per second.  Now, the system will actually queue the files up on both the agent and the appliance (up to a point and then drop) and I've seen my waiting queue up around 1.3 million events waiting to be processed which seems to be the upper limit of our resource configuration.  Still waiting to here from support on our issue.    Otherwise it works.

Definitely let me know what you hear back from support on this, I am very curious.

Not sure if you opened a ticket or called them but I have found with LEM I tend to get much better response by calling.

Thanks for chasing this down and posting it here - we're also looking at creating a sizing guide that can help with these recommendations.

A sizing guide would be most awesome!