That's a lot of syslogs. Are you using LEM for syslog retention? For extremely large amount of syslog, I have seen people send the syslog to Kiwi Syslog Server for syslog retention. Then send only the important syslogs over to LEM (or Orion). This way the company gets the 2 years of syslog retention they require while leveraging LEM for rules.
I have also seen another layer to this, which is a flowreplicator. This will allow the device(s) to send the flow (syslog, traps, netflow) to the replicator, then the replicator will send it out to whoever.
There is no explicit limit on the amount of syslog/SNMP trap volume per hour with LEM. Without any correlation rules and only storing in the raw log store, we're talking tens of thousands per second. With correlation rules and using connectors to parse the data, we're still talking hundreds on the low end to thousands per second depending on available resources (CPU, memory, disk space).
Thanks Nicole for the detail.I am planning to configure security devices to send syslog to LEM which sends 2.5millions syslog messages/hour so I am wondering whether LEM will be able to handle or not?
I am looking for any recommendation from Solarwinds on volume of acceptable messages per hour without any rules.
It's a relatively high volume, but not unheard of for LEM. With rules/alerts you'll probably have to assign more RAM/CPU. You might want to even just to collect it, but it's hard to say, if you're just storing those events the default allocations might be fine. You could likely increase that by 50-100% and still be fine.
It's look LEM can handle plenty of event. Do we have any internal tool in LEM to monitor the RAM/CPU resource rater than using Orion?
For data storage, seem LEM is using the FILO method to store the log and event. How much event or log will use 1 GB space on the storage? I know this question might be base on lots of assumption.However, having a maximum size of a event will be useful to calculate how much storage is require for my LEM for long term event storage.
You can access top under the appliance menu, when using the console with the cmc account, to get a real-time view of cpu/mem util etc, as Nicole says any longer term historic data can be obtained via vm tools etc.
Right now, there isn't an internal LEM tool - most customers are using the hypervisor to track memory/CPU usage (if you have Orion or Virtualization Manager you could track it there, too).
It's hard to calculate. The easiest way (arguably) is if you're trialing LEM to do the math based on real data: SolarWinds Knowledge Base :: How many days of live data will the LEM database store?
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.