I haven't used Orion in many years but I assume it is still backed by some SQL version. Syslog is a lot of small writes and can quickly impact the DB performance.
We have Kiwi as our primary syslog engine. It sits between our monitoring and reporting tools and NXlog, which is our 'bulk' syslog receiver. We have a pair of NXlog instances and 4 Kiwi syslog servers capturing around 1.2 million messages an hour from approximately 2200 sources.
NXlog receives all syslog traffic. It does all the writing to disk and some broad filtering. Everything that isn't filtered here is sent to Kiwi. Kiwi has many rules to match and take action on interesting data. Some of it gets written to logs, others(approximately 20%) generate emails or event logs. Items that need action(appromitely 1-2%) get sent to the monitoring tools for alerting, HD ticket generation, search, etc.
NXlog could do everything Kiwi does but Kiwi has a nice interface that is easier to get less experienced staff to understand and manage. NXlog is all done in the config file and can become hard to follow with complex matching and actions.
Deciding what it interesting data tends to be pretty specific to the organization. Generally you will want security events, hardware and application failures, non-information level events and the dreaded 'others'. With the number of sources you are talking about I would definitely use Kiwi in front of Orion.
We have seen kiwi syslog pre-filtering improve the performance in Orion environments not only from processing the log load on the pollers, but also overall DB performance. If you are properly pre-filtering messages to only actionable items, you are reducing reads and writes to the DB and in cases where syslog is noisy in an environment this can be a decent amount of data that doesn't need to reside in the Orion DB tables. If you still want that noisy data, kiwi can offload it into flat files or a DB that is not a part of your monitoring solution and is more of a data warehouse for syslog for auditing/reporting purposes.
The 2 general approaches that I have suggested in the past on getting to where you want to be with syslog monitoring using kiwi syslog server as the primary processor:
Filter known noise and forward everything just adding to the noise filtering until you reduce what is forwarded to only actionable items.
Filter know issues and forward them on, filter noise and drop it at kiwi, store all unknown content as a flat file or in a DB for review until you decide it is noise or actionable.
We have seen pushing the load off to a syslog help in several cases in client environments, so I would suggest it especially if you are considering a decent load of new nodes and are already seeing performance issues.
Thank you j_a_catlin and kstone for your replies. So, agreed, using a syslog filter should be able to assist in syslog management. The logical follow-up question becomes, "How to build a list/definition of actionable vs trivial syslog messages. I'm sure if I looked at some messages, I could determine some were actionable,and some trivial. However, there must be thousands of syslog messages, so the likelihood of me defining each one seem remote.
Is there a list of actionable (or non-actionable) syslog messages you use? I am not intending this to be a "Here, let me Google that for you." type of question. I did a google, and immediately found a resource from Cisco which has an appendix of actionably syslog messages. Or, do you just filter for severity level? What process did you go through to build your actionable syslog filter?
Many Thanks, Eric
Our default is to log everything and create additional actions for specific items.
for example, we have rules for Windows login errors, Cisco events, application errors, etc. Most of these start pretty broadly then get tuned over time.