In the first blog of this series, What are Logs?, we learned what logs are and why they are useful to your organization. Now we are going to learn what happens when you have different types of log data and how your System Administrators don’t go crazy going through the thousands of logs that are produced.
System Administrators monitor logs from hundreds of different devices, all written in proprietary formats. Unfortunately, proprietary log “language” is often not user friendly and unreadable in its native format. Log data is written in the device’s language, which is different than the “language” of another device. Comparing the logs against each other is like comparing a paragraph written in Russian, against one written in Japanese.
The following is a sample of a log file from an antivirus program:
There is some recognizable information, but can you imagine trying to decipher thousands of logs entries a day? Luckily, there are software programs (like our very own SolarWinds Log & Event Manager) that can convert that data to useful information that can be searched for important alerts or events that may have occurred.
What is Normalization?
The LEM system is based on software modules called Agents, which collect and normalize log data in real time before it’s processed by the virtual appliance, and other non-Agent devices, which send their log data directly to the Manager for both normalization and processing.
By definition, to normalize is to make (text or language) regular and consistent. LEM gathers logs from devices and translates (or normalizes) those logs into the same language, so they can be directly compared against each other.
When an Agent cannot be installed on a device, that device can be set to send its log data to the LEM Manager for normalization and processing. Examples of devices that cannot host Agent software include firewalls, routers, and other networking devices. LEM accepts normalized data and raw data from a variety of devices. Non-agent devices send their log data in raw form to the LEM manager. Once normalized, log data is processed by the LEM Manager, which provides a secure management clearinghouse for normalized data. The Manager’s policy engine correlates data based on user defined rules and local alert filters, and initiates the associated actions when applicable. These actions can include notifying users both locally in the Console and by email, blocking an IP address, shutting down or rebooting a workstation, and passing the alerts on to the LEM database for future analysis and reporting within the Reports application.
SolarWinds Log & Event Manager collects, stores, and normalizes log data from a variety of sources and displays that data in an easy to use desktop or web console for monitoring, searching, and active response.