cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

“Logfile Monitoring” – I Do Not Think That Word Means What You Think It Means

Level 17

This is a conversation I have A LOT with clients. They say we want "logfile monitoring" and I am not sure what they mean. So I end up having to unwind all the different things it COULD be, so we can get to what it is they actually need.

It's also an important clarification for me to make as SolarWinds Head Geek because depending on what the requested means, I might need to point them toward Kiwi Syslog Server, Server & Application Monitor, or Log & Event Manager (LEM).

Here’s a handy guide to identify what people are talking about. “Logfile monitoring” is usually applied to 4 different and mutually exclusive areas. Before you allow the speaker to continue, please ask them to clarify which one they are talking about:

  1. Windows Logfile
  2. Syslog
  3. Logfile aggregation
  4. Monitoring individual text files on specific servers

More clarification on each of these areas below:

Windows Logfile

Monitoring in this area refers specifically to the Windows event log, which isn’t actually a log “file” at all, but a database unique to Windows machines.

In the SolarWinds world, the tool that does this is Server & Application Monitor (SAM). Or if you are looking for a small, quick, and dirty utility, the Eventlog Forwarder for Windows will take Eventlog messages that match a search pattern and pass them via Syslog to another machine.

Syslog

Syslog is a protocol, which describes how to send a message from one machine to another on UDP port 514. The messages must fit a pre-defined structure. Syslog is different from SNMP Traps. This protocol is most often found when monitoring network and *nix (Unix, Linux) devices, although network and security devices send out their fair share as well.

In terms of products, this is covered natively by Network Performance Monitor (NPM), but as I've said often you shouldn't send syslog or trap directly to your NPM primary poller. You should send it into a syslog/trap "filtration" first. And that would be the Kiwi Syslog server (or its freeware cousin).

Logfile aggregation

This technique involves sending (or pulling) log files from multiple machines and collecting them on a central server. This collection is done at regular intervals. A second process then searches across all the collected logs, looking for trends or patterns in the enterprise. When the audit and security groups talk about “logfile monitoring,” this is usually what they mean.

As you may have already guessed, the SolarWinds tool for this job is Log & Event Manager (LEM). I should point out that LEM will ALSO receive syslog and traps, so you kind of get a twofer if you have this tool. Although, I personally STILL think you should send all of your syslog and trap to a filtration layer, and then send the non-garbage messages to the next step in the chain (NPM or LEM).

Monitoring individual text files on specific servers

This activity focuses on watching a specific (usually plain text) file in a specific directory on a specific machine, looking for a string or pattern to appear. When that pattern is found, an alert is triggered. Now it can get more involved than that—maybe not a specific file, but a file matching a specific pattern (like a date); maybe not a specific directory, but the newest sub-directory in a directory; maybe not a specific string, but a string pattern; maybe not ONE string, but 3 occurrences of the string within a 5 minute period; and so on. But the goal is the same—to find a string or pattern within a file.

Within the context of SolarWinds, Server & Application Monitor has been the go-to solution for this type of thing. But, at this moment it’s only through a series of Perl, Powershell, and VBScript templates.

We know that’s not the best way to get the job done, but that's a subject for another post.

The More You Know…

For now, it's important that you are able to clearly define—for both you and your colleagues, customers, and consumers—the difference between "logfile monitoring" and which tool or technique you need to employ to get the job done.

27 Comments
Level 12

Good post.  Loved the "Princess Bride" reference too.  Three cheers for clarity and defining of terms.

Level 17

inconcievable.jpg

This article is why you are Head Geek. Very nice breakdown. Hope you don't mind I use this as reference for future clients - Or just in general.

Level 17

I like this 'filtration system' you speak of. Scoping for next year, this gets considered for budgeting - as I see the very need for this when creating a solution that scales easier.

It will be much easier to manage when we filter out messages then aggregate.

Level 17

Mind? I'm honored and flattered. Use away!

Level 14

I always think of LEM first when considering Windows log monitoring... SAM does OK, but it's not at the same level really...

Level 17

Thanks, cause the syslog definition was just what I needed for my outline that will drive a tech user session that informs my field techs on the basics of monitoring(brief), and management (and of course the un manage process) in regards to Node Service, Maintenance, Upgrade & Refresh.

    Monitoring & Management of Nodes & Interfaces - including the use of Maintenance & Service Windows (using sql alerts)

* while typing this out I think I figured out the holes needed to fill the second session - Adjusting Management and Monitoring Pre and Post Change/Refresh to remain invisible and keep previous node/interface history.

   The use of custom properties as location indicators, and thus having a 'Not Alerting' Region to assign Nodes to has been massively helpful in turning off alerts and checks for nodes with history that we want to keep - as well as a place for Post Change Node after a migration or the refresh.  --- of course you must include this in your alerts - the custom property created should <> 'NOT ALERTING' - for nodes that you want to alert on.

Level 15

I appreciate the efforts to break this down.  I hear from so-called security gurus about the needs of logfile monitoring but they do not seem to be able to definitively define for the situation.  Also, I do find myself having to go through a similar breakdown with new juniors to get them out the correct page.  I plan to bookmark this page and use it as reference to the juniors and for security discussions.  Excellent work.

Level 15
Level 10

Simple and concise, Thanks!

Level 7
Level 13

I often wonder what the use case would be for Kiwi Syslog Server when I already have NPM. The filtration concept makes perfect sense, and I'm slightly embarrassed to admit that I hadn't thought of it before. Great idea!

Level 17

If you have junior monitoring folks, you may also want to take a look at this: Monitoring 101

Level 10

This is a great article!

MVP
MVP

Thanks adamlboyd‌‌ for sharing the link. This is exactly what we were talking about just this morning. We already have this: "send all of your syslog and trap to a filtration layer" covered so maybe we should go the LEM route.

And thanks adatole‌ for the write up. It's really a great break down of which product we need to be looking at.

MVP
MVP

Monitoring individual text files on specific servers - without using perl, powershell, vbscript is the area that sorely needs to be addressed.

Having to watch for multiple strings to cover different events using home grown solutions executed remotely is still inefficient at best. 

Level 15

I would add that I am being asked more and more to incorporate into our logging data from API sources.

Level 16

Over the many years I have been using NPM/SAM/LEM I have been waiting and hoping for better individual text file monitoring.  I NEED this in LEM for many systems that create a custom log file in basic text that would need to be alerted on and saved for future troubleshooting.

Windows logs are handled by LEM and SAM very well and we are very pleased with that.

Syslogs and traps also are handled well with LEM and if you have a reasonable quantity NPM does a decent job, although I would LOVE/NEED to see the alerting updated/incorporated with the web alerting.

Great post adatole‌ !!

MVP
MVP

It is fun...NOT...monitoring multiple instances (3 - 😎 of the same log file (multi-tenant system) for 12 specific message strings across 4 - 8 servers.  This ends up being a huge overhead.

Level 16

I hear that!  We have hundreds of log files that I need to get in LEM but no way to do that just yet.  I am relying on my feature requests!

Good article, help define the different types of logs for those new to monitoring. 

Last year we rolled out Solarwinds LEM.

Did you guys know that EVERYTHING creates logs? Yea, I did too. But there is...

SO.MUCH.LOGS!!!

The early stages of LEM was like my first foray into JonFacienda narrated NFL Flims clips on YouTube. It never stops...

Level 8

So we're now near the end of 2016... I'm still waiting for LEM to be able to Monitor individual text files on specific servers which happen to be application log files! 

Custom templates on SAM for every application text log is PAINFUL!

MVP
MVP

especially if the log is chatty....very chatty.

Powershell is not a viable solution to watch for many different specific strings in a large busy text logfile for most people.

Hello,

could you please help me with process or tool that would help me monitor specific text in a log\txt file and alert me.

Level 13

If you're on Linux/Unix, tripwire is probably the most popular.  Easy to tune to filter out whatever you're looking for as well.  Redhat ships with a logwatch tool as well that will email you daily with default important events.

Level 12

thanks for the post

Level 17

there are a few SAM templates in the content exchange (most authored by aLTeReGo​) that do this.

About the Author
In my sordid career, I have been an actor, bug exterminator and wild-animal remover (nothing crazy like pumas or wildebeasts. Just skunks and raccoons.), electrician, carpenter, stage-combat instructor, American Sign Language interpreter, and Sunday school teacher. Oh, and I work with computers. Since 1989 (when you got a free copy of Windows 286 on twelve 5¼” floppies when you bought a copy of Excel 1.0) I have worked as a classroom instructor, courseware designer, desktop support tech, server support engineer, and software distribution expert. Then about 14 years ago I got involved with systems monitoring. I've worked with a wide range of tools: Tivoli, Nagios, Patrol, ZenOss, OpenView, SiteScope, and of course SolarWinds. I've designed solutions for companies that were extremely modest (~10 systems) to those that were mind-bogglingly large (250,000 systems in 5,000 locations). During that time, I've had to chance to learn about monitoring all types of systems – routers, switches, load-balancers, and SAN fabric as well as windows, linux, and unix servers running on physical and virtual platforms.