Log retention and disk space

Hey there,

The amount of storage in days is of course related to the size of stuff you get per day. No two deployments are alike which makes a number hard to predict, but since you're deployed we can learn about your data empirically and make some extrapolations. On to the science...

That disk space (what you're most interested in is the "Logs/Data" partition) is shared among:

1. Syslog/SNMP data being directly received on the appliance
2. The normalized database store
   
3. The original/raw message store, if enabled
   

The easiest way to see how far back your database data (#2) goes is with a "Database Maintenance Report" in LEM Reports. If you've enabled storage for original log data, you can also determine how far back your original log store (#3) goes back with a "Log Storage Maintenance Report".The appliance will store 50 days of the directly received syslog/SNMP data, rotated and compressed daily. (The data itself is read in real-time and processed, it's kept around in case you want access to the original syslog data without using our dedicated original log storage.)

Your "Logs" number in the diskusage above indicates the total value of all received syslog/SNMP data comprises 62G of the 86G of data (from "Logs/Data"). This will hit approximate steady state once you get to that 50 day mark, and generally stop consuming more space unless you have an abnormal influx of messages or add more syslog/SNMP devices. The remaining 14G of data is your actual database (and original log store, if enabled), which makes sense - the database gets pretty good compression in general.

Given that a significant portion of your disk is syslog data, you could deploy a second LEM appliance and dedicate it to syslog processing. We can set the second appliance up so that all the processed syslog data is sent to the LEM management appliance (for storage, correlation, etc). We can actually do the same thing with the database, assign it to a dedicated appliance; in that case, the management appliance will store the data on the database appliance. Or, you could do both! ;)

We're also researching the best way to expand the disk, in cases where people are willing to dedicate more space to store more data in the all-in-one appliance.

Let us know if you want more help with the math based on your specific data - our support team is top notch (think more Mythbusters, less Bunsen Honeydew).

I have LEM and need to meet the following requirements 6 month retention for log files

profzoom1, what are your average write and read IOPS on your LEM appliance? Also, how quickly do reports complete for you on average?