This is the third post in a five-part series uncovering the mysteries of logs. In the first two posts (What Are Logs? and Logs: So Many Different Types), we discussed the various uses for, and types and formats of logs. In this post, we'll complicate even further by looking at how each of the different types of logs support their own unique methods for getting at their valuable data.
How to Collect Logs
Now that we're familiar with the four general types of logs, let's take a peek at how to get at them. In all but one case, each type of logs supports both a "pull" and "push" option for log retrieval. Think of the "pull" option as you going somewhere to get the logs, and the "push" option as the system, device, or application sending the logs to you.
Windows Event Logs
- Pull - The traditional and most common method for retrieving Windows Event Logs is through the Event Viewer snap-in for the Microsoft Management Console (MMC) on the local system. On the surface, it seems like this would require you to remote into each system for which you want to view logs, but you can also connect the Event Viewer snap-in on your local machine to remote systems. I discussed this option in a bit more detail in another post: Get all of your Windows management tools in a single pane of glass.
- Push - Starting with Windows Server 2008, Windows also provides a method by which you can make a system "publish" its logs, allowing you to "subscribe" to them. If you're managing older systems, the "pull" method is really your only option without bringing a third-party log aggregator into the mix. One example is the free Event Log Consolidator from SolarWinds.
- Pull - The "pull" method for text logs is pretty rudimentary: Open the file. Sometimes this isn't as simple as it sounds, though. For applications, this could be as easy as opening a file. For network devices, it probably involves logging into a command-line interface and running a series of commands. In either case, it's important to revisit a point made in Part 1: Just because a system uses the "text" logging type, that doesn't mean you can read it. In many cases, it takes years of practice to be fluent in the language of your logs.
- Push - The most common "push" method for text logs is syslog. This method requires the logging device to both translate (or create) its logs into the syslog format, and then transmit the logs via the syslog protocol to a server the administrator specifies. From a log management perspective, if your devices and systems provide a syslog option for their logs, sending your logs to a dedicated syslog server is one of the most effective ways you can streamline your process.
- Pull - When SNMP is enabled on a device, it stores its data locally, waiting for something to poll it. This retrieval method is a little less common from a logging perspective, but it's the best-known implementation of SNMP as a protocol.
- Push - This method involves one of the most confusing terms in network management history (IMNSHO). An SNMP trap is a message that an SNMP-enabled device sends out when a certain threshold is met. For this to work, you have to define the thresholds and destination for the device. Historically, network devices and some antivirus vendors are what use the SNMP trap method for logging and state/status notifications.
This is the one category that really only supports one retrieval method: pull. When a system stores its logs in a database, they're there until you ask for them. In other words, you have to query the database. You might do this through a pretty web interface or a plain old SQL management tool, but in either case a database query is involved.
All That Trouble Just for Logs?
After learning about the various types, formats, and retrieval methods for log data, you might be wondering why you'd ever want to go through so much trouble to get at your logs. Well, remember from Part 1 that logs can be very useful. Luckily for the troubleshooting use-case, Windows logs are relatively easy to get at and most network devices support syslog. But if you want to take a more comprehensive approach - whether for compliance reporting or proactive analysis - you'll probably want to call in the help of some experts. While your organization might not be able to afford (or justify) a dedicated security specialist to collect and analyze your logs, it might be worth taking a look at a dedicated log management system.
In the next installment, we'll look at yet another variable in this intriguing equation: Reading the Logs.