I'm trying to figure out on average what is volume of logs we're bringing in on a daily basis. Am I bringing in 2Gb of data on a daily basis or am I bringing in 800mbs a day I have no idea.
So far I haven't found an easy way to do this. Support said to run the database maintenance report but that only shows me the number of events that were brought in not the size on that given day.
I then SSH to the box and ran the diskusage command. I got the report below. Now the Logs: 829M stood out to me and it actually will increase every minute or so. I'm assuming that is what I'm looking for and if so I'm just looking for some verification. Then I can just monitor this to get an idea at the end of the day around midnight to see what I'm averaging for volume of logs coming in.
So if Anyone has any thoughts on if I'm right that this is the correct thing to be looking at or not and if you can point me in the right direction.
Partition Disk Usage:
LEM: 54% (1.5G/2.9G)
OS: 41% (1.2G/2.9G)
Logs/Data: 57% (123G/230G)
Temp: 19% (1013M/5.8G)
Database Queue(s): 4.0K (No alerts queued, -8257239616 alerts waiting in memory)
Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
DataCenter Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
Forensic Database Queue: 2.1M (0 data queued, 0 data items waiting in memory)
Logs: 829M <========= Volume of logs at this point of the day?
Tool Profiles Message Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
Logs reported in that part of diskusage is direct syslog messages sent, so it won't account for what's actually stored in the LEM database (i.e. all the other stuff sent from agents and indirect logging). The archive that Pradeep refers to is the actual database storage, so that would tell you what your true event storage looks like. This is the "Data" part of Logs/Data (the other 122G of your 123G).
You might also want to check out the Database Maintenance Report on the reports side, there's a couple of related reports there that do try to break down your events and database growth. That might help paint a different picture.
Here's how I do it:
1) Configure the LEM appliance to do a daily backup for the logs - archiveconfig option under cmc->manager. This will kick off at 6:25AM daily.
2) Then it's a simple matter of browsing to the network share you configured archiveconfig to point to with Windows Explorer, go into the SolarWindsLEMAlertDBArchive subfolder, and sort by date. Highlight/select the .dat files for the day in question, right click->Properties, and you have the total volume for that day (or more specifically the 24 hours thru to 6:25AM) (note we aren't capturing raw logs in our environment so not sure if those are stored somewhere else).
3) Keep in mind the first time you run archiveconfig it will do a full backup, which could take a while. Subsequent runs will just do an incremental backup, and give you the data storage for the previous 24 hours.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.