If you’re using network monitoring solutions (regardless if it is an enterprise size or designed rather for smaller environments) to digest log messages from your network devices, sooner or later you’ll likely notice some performance issues caused by the hundred-thousands (or even millions) of syslog messages and SNMP traps devices can generate per hour, which can overload your monitoring server. Ingesting so many syslog messages and SNMP traps can negatively affect standard monitoring operations, such as polling devices, collecting log files, or running alert queries. At the same time, a majority of these logs don’t have monitoring impact and create “noise.” Plus, your monitoring solution may be priced based on number of processed logs, making this not only a performance, but a budget issue as well.
But what can you do about it? Maybe you could remove some of the chattiest devices producing only “noise” most of the time? Security policies of your company most likely require you to store all your log data, so this is a no-go for your security team. Plus, even from these devices, you still want to be sure you don’t miss messages with higher severity levels, right?
In fact, this is rather common scenario for a lot for larger environments. Fortunately, it has quite a neat solution.
One easy solution can be to place a simple, open-source or very low-cost syslog server on separate machine that will receive SNMP traps and syslog messages and SNMP traps and act as a filtration layer for all logs your network devices create. To make it work, you’ll need to reconfigure your device to send it to this syslog server. In larger environments, it doesn’t have to be only one server, but a series of servers behind a load balancer which uses round-robin, so you can add as many syslog servers (receivers) as the size of your environment requires. Here, you apply the rules to do the filtering, while storing the logs and SNMP traps in a database for you audit team or later use. Only logs with monitoring impact will be sent to your monitoring solution, reducing the number of logs by at least 50% (but probably way more). One thing you need to keep it mind is the syslog server you’re using as your “filtration layer” needs to be able to preserve the hostname of the original message when forwarding it to another place, so you don’t lose this essential information.
Now, you have a perfectly optimized system with effective filtration layer that relieves the pressure from your monitoring solution and avoids headaches associated with this essential aspect of monitoring. Only actionable messages (such as routing neighbor down…) are forwarded to your monitoring solution, where they’re associated with other data your monitoring solutions collects and alerts you in case you need to fix something.
If you’re looking for a syslog server to help you put this filtration system in place, try free trial SolarWinds® Kiwi Syslog® Server. Kiwi Syslog Server is a simple log management solution that allows you to centralize your syslog messages, SNMP traps, and even Windows event logs. You can also use Kiwi Syslog Server to apply rules and forward selected messages to another solution while keeping your logs archived for compliance purposes.