cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Log analysis tools review: syslog, filtering the most relevant events

Level 9

Who’s passed more than a weekend going almost blind because something in your <replace with your beloved system, not necessarily IT> didn't work as it should, and it produced, hopefully, thousands of lines of error messages, almost all of which were the same? The key words here are, “almost the same."

A syslog system could save you precious time, and your family will be thankful for it, but it's up to you to find a way to filter the lines that are "almost" like the others, the ones that are different save for a single cypher that makes your day.

Syslog is a tool. Actually, it is the first logging tool. Try to put a screwdriver beside your PC, you won't be disappointed if it doesn't disassemble the machine. The tool leverages your skills, it doesn’t replace them.

There are several tricks to filter the lines produced by a syslog. The key is to find the right pattern.

Today there are so many GUI tools to perform these operations, but the core is the same: match a pattern to the bundle of lines, and filter the ones that will help to solve your problem.

Taking a step back, during syslog configuration, you can set it up to write the logs in a manner that is closer to your pattern’s creation.

We could ask to put the time-data at the beginning of the line, or maybe the severity, or, again, the service involved. That's not important. What is important is that you do it in a way that suits you.

So, the pattern. This is the critical point, the difference between success or wasted nights. Similar as a Google research, you need to find the right keyword for the best result.

There are several free tools to parse logs, and the same goes for patterns. There are many databases of them, but maybe your issue is more specific? Anyway, a good starting point is log parsing.

Today there are many advanced utilities born from syslog, a number of which are open source: rsyslog, syslog-ng, logwatch, just to name a few. The main difference is that rsyslog apply filters on the logs produced by syslog to perform actions, for example, if present the word "localhost" sends an email to someone@domain.com, or if the source IP 10.10.10.10 is present it writes the line on file "10101010".

Syslog-ng is a more complex utility that not only filters, but also correlates and classifies. Logwatch is interesting, and I’ll cover it in a future post.

All of the tools use a set of precompiled patterns, in many cases modifiable. Let me say that it's quite unusual to not find the right filter for your very specific requirements.

Besides these tools there's another subset to consider when talking of logs: SNMP - traps and polling. Usually they're used for monitoring purposes instead of analysis, but the core concept is the same: a tool that writes lines that are constantly filtered by another tool that sends traps - or waits to be read by a poller to raise an alert. The first part of the process is the same: logging - and the second is similar too: filtering.

So, enjoy, try, install, reinstall, destroy, but above all, keep logging! It can save you a lot of precious free time, and it can help your peers as well, even those from other parts of the world.

13 Comments
Level 16

I set ours up like this:    Syslog source (Router/Switch/AP)  ---> Kiwi Syslog Server ---> Orion NPM   ----> Alerts

Kiwi can pre-filter and send only the important ones to Orion NPM to make alerts on.

Level 14

When I got here I found a Win2k3 syslog server running Kiwi but no one was using it.  It's now a Win2k8 server still running Kiwi and no one using it.  Baby steps.

Level 13

Good Article - been looking at this for one of our customers recently.

Level 16

We receive about half of our Cisco alerts via Syslog and the others are polled through NPM. Occasionally some network event will trigger an event storm so large that it would start to bury the NPM server's database. That's why we front ended it with Kiwi

to do the filtering and then only forward on specific events.

MVP
MVP

Finding out recently we may need to keep 5 years of logs instead of 1 ugh!

Level 16

We were just doing cost analysis for our existing system. Creating a calculator that will determine the cost per 1gb log ingestion per day, having it 'hot' searchable for 90 days, frozen 91 days - 365 days (1 year total). Then cost per 1gb frozen per year going forward.

Almost done

We are still tackling the nuances of syslog. Not all are the same. Different OS'es and apps require different customisation. Thus different monitoring rules.

Level 9

Exactly - I'm just writing a post about customisation, I think that this feature, together with filtering, could be the key of success in monitoring and auditing.

Level 9

Thank you David. Hope it was useful.

MVP
MVP

Lot's of parts and pieces of the Log puzzle. What to log, what to keep, what to alert on, how to log, how to review, where to store things, how to store things. But the biggest of all is who is going to consume that information and turn it into value. Logging is necessitated by audits, laws and regulations, but using that data is where the company benefits. Sometimes it's in finding "bad actors" and correcting things, other times it's finding inefficiencies in processes or equipment, and sometimes it's identifying waste and saving the company money.

Lot's of good reasons for logging, but take the business case to management and find creative ways to actually use this data.

Level 9

I completely agree: logs are just data. The added value is the way this data are used. And if an admin is able to shape this flow for special purposes inside his company, well, this is a very good reason why "things keep logging"!

Level 21

Five years ago I would have totally agreed with all of the points here but things have changed and so has my perspective on this.  While I think managing your logs is great, just basic syslog and alerting will no longer cut it for environments of any size due to the sheer volume of logs you will end up dealing with.  Syslog is certainly a great place to start but I think most folks will quickly outgrow a simple Syslog solution and need something more advanced with more advanced parsing, dash-boarding and reporting capabilities.  When dealing with large quantities of logs I have found alerting almost unruly, in a few cases for specific events it can work out well but overall it becomes unmanageable.  What I have found works out best is having a more advanced log management system or even SIEM that is capable of processing the events and turning them into dashboards that provide business and/or operational insights that are then reviewed either constantly or multiple times a day by folks that have been trained on what to look for.   

Level 9

You're right.

Actually, my piece was just the beginning, as you stated, a great place to start. Then, for sure today syslog isn't enough, but anyone dealing with logs should now this tool, in my opinion. And then move to more sophisticated tools.