We use Orion NPM syslog as the primary collector, but many devices have not been configured to send syslog to Orion yet. Recently, I was talking with the Director of IT I report up through regarding syslog. He was thinking that were we to increase our syslog sources by a thousand or so nodes, that doing so would cause Orion to slow down ore than it already is. What do you think of that point of view? Would it be better to off-load collection duties to another syslog collector?
We have a licensed version of Kiwi Syslog, which we have been using on a limited basis (and separate from our Orion installation). I have heard that some people use Kiwi Syslog as their primary syslog collection software, instead of using Orion NPM syslog as primary. What are the advantages and drawbacks you have experienced in using this method? I have heard that you can't view Kiwi Syslog logs from the Orion GUI. But, I have also heard that Kiwi can forward interesting log entries from it to Orion. However, that brings up a question, of how do you define interesting log entries for forwarding? What may not normally be interesting to some could be interesting to others, depending on what event I am looking for.
Anyway, if you use Kiwi Syslog Server as your primary collector, could you please share your input here?
We have seen kiwi syslog pre-filtering improve the performance in Orion environments not only from processing the log load on the pollers, but also overall DB performance. If you are properly pre-filtering messages to only actionable items, you are reducing reads and writes to the DB and in cases where syslog is noisy in an environment this can be a decent amount of data that doesn't need to reside in the Orion DB tables. If you still want that noisy data, kiwi can offload it into flat files or a DB that is not a part of your monitoring solution and is more of a data warehouse for syslog for auditing/reporting purposes.
The 2 general approaches that I have suggested in the past on getting to where you want to be with syslog monitoring using kiwi syslog server as the primary processor:
Filter known noise and forward everything just adding to the noise filtering until you reduce what is forwarded to only actionable items.
Filter know issues and forward them on, filter noise and drop it at kiwi, store all unknown content as a flat file or in a DB for review until you decide it is noise or actionable.
We have seen pushing the load off to a syslog help in several cases in client environments, so I would suggest it especially if you are considering a decent load of new nodes and are already seeing performance issues.
Thank you j_a_catlin and kstone for your replies. So, agreed, using a syslog filter should be able to assist in syslog management. The logical follow-up question becomes, "How to build a list/definition of actionable vs trivial syslog messages. I'm sure if I looked at some messages, I could determine some were actionable,and some trivial. However, there must be thousands of syslog messages, so the likelihood of me defining each one seem remote.
Is there a list of actionable (or non-actionable) syslog messages you use? I am not intending this to be a "Here, let me Google that for you." type of question. I did a google, and immediately found a resource from Cisco which has an appendix of actionably syslog messages. Or, do you just filter for severity level? What process did you go through to build your actionable syslog filter?
Many Thanks, Eric
Our default is to log everything and create additional actions for specific items.
for example, we have rules for Windows login errors, Cisco events, application errors, etc. Most of these start pretty broadly then get tuned over time.
I haven't used Orion in many years but I assume it is still backed by some SQL version. Syslog is a lot of small writes and can quickly impact the DB performance.
We have Kiwi as our primary syslog engine. It sits between our monitoring and reporting tools and NXlog, which is our 'bulk' syslog receiver. We have a pair of NXlog instances and 4 Kiwi syslog servers capturing around 1.2 million messages an hour from approximately 2200 sources.
NXlog receives all syslog traffic. It does all the writing to disk and some broad filtering. Everything that isn't filtered here is sent to Kiwi. Kiwi has many rules to match and take action on interesting data. Some of it gets written to logs, others(approximately 20%) generate emails or event logs. Items that need action(appromitely 1-2%) get sent to the monitoring tools for alerting, HD ticket generation, search, etc.
NXlog could do everything Kiwi does but Kiwi has a nice interface that is easier to get less experienced staff to understand and manage. NXlog is all done in the config file and can become hard to follow with complex matching and actions.
Deciding what it interesting data tends to be pretty specific to the organization. Generally you will want security events, hardware and application failures, non-information level events and the dreaded 'others'. With the number of sources you are talking about I would definitely use Kiwi in front of Orion.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.