Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Proper Syslog DB management

We have a requirement to keep all Syslog messages for 365 days, so obviously that causes an issue on our DB size. Which in turn causes issues with searchability and even how the daemon itself runs.

We are looking for advice on how to properly manage that DB so we meet our requirement, but can still search the logs either from the web interface or the app on the server. I'm not a DBA and have no clue on proper management techniques.

We were already a Kiwi CatTools customer, although we haven't used that Syslog server in a while. Is it better to migrate back to the Kiwi Syslog to go back to flat files and is there a way to search that syslog via the Orion Web interface?

Any assistance you can offer will be greatly appreciated.

Thank you.

Find more posts tagged with

Accepted answers

All comments

chris.lapoint

I'm really interested in your use-case for data retention and search, so a couple of questions for you before I try to provide any recommendations:

Is 365 days being driven by internal policy, regulatory requirement, or an audit requirement?
How large does your database grow with 365 days of data before you roll this data out? For example, does it grow from 8GB to 1TB? If you need to extrapolate expected database size based on a week's worth of data (multiplied by 52 weeks), that would be fine.
Would it be okay to search all 365 days via desktop application only and search a subset using web interface? If so, what would be acceptable subset for web interface (e.g. last 30 days)?

FormerMember

1. All three apply. It is an internal policy, requirement and needed for audits.

2. We currently are retaining 150 days on the server although our requirement is 365. We made a "group" decision about that far back to cut it to 150 days and if we needed records older than that, we would retrieve them from tape. We are just now hitting about that 150 days worth of retained data and the current DB size is 29G. This is also a shared DB with other applications, with approximately 50G of free space on the drive. However the SolarWinds DB is currently at 29G.

3. If we could search 30-45 days through the web app and then resort to the Syslog Viewer on the server itself for anything older, that would be great. However my issue currently is, I can't even get the Syslog Viewer to even function. It goes to "Not Responding" about 15 seconds after being launched.

Hope I answered your questions.

Thank you,

chris.lapoint

Thanks for answering those questions. Based on you requirements, here's what I would suggest doing:

Deploy Kiwi Syslog Server as a transparent proxy for Orion Syslog. Kiwi Syslog Server logs all data to the file system so you don't need to worry about any database (just disk space). It also provides the ability to archive/compress data on a scheduled basis, which will likely managing that much data a lot easier.
Configure Kiwi Syslog Server to forward syslog to the Orion Syslog Server for viewing/searching within the Orion web console. If possible, you might configure Kiwi Syslog Server to filter the events to only forward the critical operations-related syslog events to Orion Syslog Server. Kiwi Syslog Server provides the ability to retain the original source IP in proxy mode.
Configure Orion Syslog Server (File > Settings) to retain Syslog data for only 45 days. This should reduce the strain on the DB and the Orion Syslog Viewer.
Use Kiwi Log Viewer to view/search old Kiwi Syslog files if you need to go back to review historical data or simply need to check a box on an audit/regulatory requirement for this capability

Hopefully, this all makes sense. If not, you know where to find me ;-)