cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Syslog - The Blue-Collar Worker Of The Data Center

Level 9

Striking a balance between good visibility into infrastructure events and too much noise is difficult. I’ve worked in plenty of enterprise environments where multiple tools are deployed (often covering the same infrastructure elements) to monitor for critical infrastructure events. They are frequently deployed with minimal customization and seen as a pain to deal with by the teams who aren’t directly responsible for them. They’re infrequently updated, and it’s a lot of hard work to get all of the elements of your infrastructure covered. Some of this is due to the tools being difficult to deploy, and some of it is due to a lack of available resources, but mostly the problem is people.

Information security is a similar beast to deal with. There are many tools available to help monitor your environment for security breaches and critical issues, and lots of data centers with multiple tools installed. Yet, enterprises continue to suffer from having poor visibility into their environments. 

Syslog is an underrated tool. I call it the blue-collar worker of the DC. It will happily sit in your environment and collect your log data for you, ready to dish up information about problems and respond to issues at a moment’s notice. All you have to do is tell your devices to send information in its direction and it takes care of the rest. It catches the messages, stores them, analyzes them, and sends you information when there’s something you should probably look at.

But it’s not just a useful way to view events in your DC. It can also be a very useful tool when it comes to managing security incidents. 

First, set up sensible alerting in your syslog environment. An important consideration is understanding what you want to alert on. Turn on too much information and the alerts will get sent to someone’s trash file rather than read. Turn on too little and you won’t catch the minor issues before they become major ones. You should also think about the right mechanism to use for alerting. Some people work well with email, while others prefer messages or dashboard access.

The second thing is to understand what you want to look for in your events. Events such as logins using local credentials when your devices use directory services are an example of something that should trigger an alert. The key is to understand what’s normal in your environment and alert on things that aren’t. It can take some time to understand what is normal, but it’s worth the effort.

Third, you must understand how to respond when an alert comes your way. Syslog is a great tool to have running in the DC because it centralizes your infrastructure logging and provides a single place to look when things go awry. But you still need to evaluate the severity of the problem, the available resolutions, and the impact on the business.

The key to managing security events with syslog is to have the right information at hand. Syslog gives you the information you need in terms of what, when, and who. It can’t always tell you how it happened, but having that information in one place makes working that out easier.

Infrastructure operations can be challenging, particularly when there’s been some kind of security incident. Having the right tools in place gives you a better chance of getting through those events without too many problems and your sanity intact.

17 Comments

I have a standing rule that every device have syslog messages, and SNMP Traps, sent to our NPM (I haven't been funded for a SIEM yet). Don't worry, very few of our techs follow it. Just about everything, from the hardware to the application, has the capability to generate messages. And just about each device generates their own unique version so the customization for alerting is a chore.

Level 16

We have a pretty large logging system and we write up a lot of exclusion rules and de-duplication rules before creating the alerts.  

I like the topic.  Particularly when earlier Thwack stories have had the slant that syslog is antiquated and obsolete, and should be replaced.  Syslog plays a very valuable role in my organization.

Level 13

Good Article. You also need to look at how long to keep logs.

Level 16

Syslog isn't going away. A lot of our devices like badge readers, medical devices, etc run on a Linux core so Syslog is the default facility they use for logging. As the need to log items such as who's accessing a data closet, entering a data center, etc for compliance reasons I only see the usage increasing. On average our logging has increased triple fold over the past couple of years and it has been driven by the need to use the log data for specific applications. Initially we are logging to SSD storage then after a few months we move it to mid tier spinning disk, then after a year its moved to the even cheaper (and slower) disk but is still there to query. Overall the data from our network devices is only about 10 percent of our overall total and that percentage is getting smaller all of the time as more applications are brought into the system.

Currently I am beginning to port all of our SNMP traffic into our logging system as well. 

MVP
MVP

Good article

Level 9

Hi Peter - that's a really good point. The amount of times I've tried to get things added in only to be thwarted by the effort required is disheartening at times.

Level 9

That's a very good point David. Unlike a puppy for Christmas, device logs are not necessarily forever. 

Level 9

Thank you!

MVP
MVP

Nice article. Revamping how we are logging is on the list for this year.

Level 8

As someone who setup a lot of our kiwi syslog captures, I can't agree more with this article. It's relatively easy to setup and customizable for your needs. For large environments the auto-split makes it convenient as new  log files can be generated as you add hosts to your network. The alerts also make it pretty straight forward for an ops environment. Sounding audible alarms and changing text to red or green based on the bad or good changes in your environment. Sadly...I still see a lot of people login to individual hosts and run "show log"....

Level 11

Great article. As a network administrator, you live for syslogs.... Why, because it gives you great insight into what is going on with those particular devices and will also provide AAA information. If you are granted a SIEM suite, even better, to be able to search for particular events that would provide clues into data breaches or security anamolies. Syslogs are very important to network engineers to troubleshoot problems too. No question about it, it is a VALUABLE tool

Level 21

I just responded to THIS post regarding Syslog and I feel that response is totally relevant here also:

While I think managing your logs is great, just basic syslog and alerting will no longer cut it for environments of any size due to the sheer volume of logs you will end up dealing with.  Syslog is certainly a great place to start but I think most folks will quickly outgrow a simple Syslog solution and need something more advanced with more advanced parsing, dash-boarding and reporting capabilities.  When dealing with large quantities of logs I have found alerting almost unruly, in a few cases for specific events it can work out well but overall it becomes unmanageable.  What I have found works out best is having a more advanced log management system or even SIEM that is capable of processing the events and turning them into dashboards that provide business and/or operational insights that are then reviewed either constantly or multiple times a day by folks that have been trained on what to look for.

From a security approach a great way to start is using a Threat Matrix where you start with the different threats you are looking for, then you note what data you need to identify the threat, then how you go about detecting such a threat and lastly what you do when a threat is detected.  This approach is product agnostic and provides a structure for you to build your technology and training around.  It also gives your analysts or threat hunters prescriptive paths to follow.  Keep in mind that this is a living document that is always being expanded and modified as things change.

Level 9

Is there a general rule of thumb, regarding how much data Kiwi Syslog can gather/filter in a day?  I'd like to push more stuff to that server, but I feel like I might be pressing my luck if I send too much more traffic/data to it.

Level 21

thwackerinva​ That information is documented HERE.  I hope this helps!

Level 16

2 million per hour was about the breaking point on mine. It depends on if you are logging them to Disk or not.

Level 16

I was logging way more than they state before mine would break. All of my messages logged to a file on disk. Try to keep it local disk if possible, not SAN and it will do better.

pastedImage_0.png