“Logfile Monitoring” – I Do Not Think That Word Means What You Think It Means
This is a conversation I have A LOT with clients. They say we want "logfile monitoring" and I am not sure what they mean. So I end up having to unwind all the different things it COULD be, so we can get to what it is they actually need.
It's also an important clarification for me to make as SolarWinds Head Geek because depending on what the requested means, I might need to point them toward Kiwi Syslog Server, Server & Application Monitor, or Log & Event Manager (LEM).
Here’s a handy guide to identify what people are talking about. “Logfile monitoring” is usually applied to 4 different and mutually exclusive areas. Before you allow the speaker to continue, please ask them to clarify which one they are talking about:
- Windows Logfile
- Syslog
- Logfile aggregation
- Monitoring individual text files on specific servers
More clarification on each of these areas below:
Windows Logfile
Monitoring in this area refers specifically to the Windows event log, which isn’t actually a log “file” at all, but a database unique to Windows machines.
In the SolarWinds world, the tool that does this is Server & Application Monitor (SAM). Or if you are looking for a small, quick, and dirty utility, the Eventlog Forwarder for Windows will take Eventlog messages that match a search pattern and pass them via Syslog to another machine.
Syslog
Syslog is a protocol, which describes how to send a message from one machine to another on UDP port 514. The messages must fit a pre-defined structure. Syslog is different from SNMP Traps. This protocol is most often found when monitoring network and *nix (Unix, Linux) devices, although network and security devices send out their fair share as well.
In terms of products, this is covered natively by Network Performance Monitor (NPM), but as I've said often you shouldn't send syslog or trap directly to your NPM primary poller. You should send it into a syslog/trap "filtration" first. And that would be the Kiwi Syslog server (or its freeware cousin).
Logfile aggregation
This technique involves sending (or pulling) log files from multiple machines and collecting them on a central server. This collection is done at regular intervals. A second process then searches across all the collected logs, looking for trends or patterns in the enterprise. When the audit and security groups talk about “logfile monitoring,” this is usually what they mean.
As you may have already guessed, the SolarWinds tool for this job is Log & Event Manager (LEM). I should point out that LEM will ALSO receive syslog and traps, so you kind of get a twofer if you have this tool. Although, I personally STILL think you should send all of your syslog and trap to a filtration layer, and then send the non-garbage messages to the next step in the chain (NPM or LEM).
Monitoring individual text files on specific servers
This activity focuses on watching a specific (usually plain text) file in a specific directory on a specific machine, looking for a string or pattern to appear. When that pattern is found, an alert is triggered. Now it can get more involved than that—maybe not a specific file, but a file matching a specific pattern (like a date); maybe not a specific directory, but the newest sub-directory in a directory; maybe not a specific string, but a string pattern; maybe not ONE string, but 3 occurrences of the string within a 5 minute period; and so on. But the goal is the same—to find a string or pattern within a file.
Within the context of SolarWinds, Server & Application Monitor has been the go-to solution for this type of thing. But, at this moment it’s only through a series of Perl, Powershell, and VBScript templates.
We know that’s not the best way to get the job done, but that's a subject for another post.
The More You Know…
For now, it's important that you are able to clearly define—for both you and your colleagues, customers, and consumers—the difference between "logfile monitoring" and which tool or technique you need to employ to get the job done.
Top Comments