Best Practices for Log Management

Version 1

    With rob.johnson

     

    Having worked with log management and SIEM (Security Information and Event Management) for several years, I have learned how critical it is to create a log collection plan and determine what you need from your logs.

     

    It doesn’t matter if you’re a dedicated Security Engineer, a Network Architect, or a System Admin, collecting logs is a crucial part of any successful security strategy.

     

    Log management really boils down to one question: which logs should you collect? To answer this question, here are some questions that I recommend asking as well as other best practice recommendations:

    1. How will log collection and storage impact your system resources? Whether it’s a free Syslog Server you spin up on a Linux® VM or a full-blown log management solution, you will need to provide system resources (i.e. memory, CPU, and storage).
      1. Storage is almost always the most expensive resource. Make sure you have a detailed understanding of how your logging solution interacts with your storage resources and whether there is any compression that is either configurable or automated. 
      2. Memory and CPU will take the next hit. Real-time collection tends to use more sustained memory and CPU. The amounts vary based on volume because log collection and storage occur simultaneously.
      3. Review your documentation to understand the impact logging will have on your devices, servers, and applications once it’s enabled. 
    2. Do you have any specific requirements for logging? Compliance regulations usually drive this conversation; however, you may also have specific internal requirements for legal or security purposes. Either way, a detailed understanding of any requirement ensures that you collect the right information from the right devices.
      1. For example: PCI provides some detail on what should be collected from your audit logs. See page 56 of this document for a complete list of details: https://www.pcisecuritystandards.org/documents/pci_dss_v2.pdf
    3. Understand what information your systems log, and how. As I mentioned in the previous suggestion, the amount of log information generated by your different devices, operating systems, and applications can be enormous. So, along with determining the actual requirements for which logs you need to collect, it is important to understand how each different system generates logs.
      1. For example, Windows® operating systems offer a detailed audit configuration that allows you to be selective about which logs are generated (i.e. logons/logoffs, privileged use, system events, and others). Additionally, you can choose whether to use successful or failed events, which allows you to take a more granular approach to log configuration.
    4. Schedule and secure archives. Data compression rates have grown significantly over the years, allowing for longer time frames for available data. However, archiving logs is still a critical function of log management.
      1. Before you schedule archives, make sure you understand the compression ratio of the archive. This will be especially important as your log store grows compared to the amount of your available archive space.
      2. Become familiar with how your log management system archives data (i.e. by device? entire database? specific logs?). Understanding the available methods of archiving saves you time, and more importantly, storage space.
      3. Secure your archives, even if it’s not a regulatory requirement. You can secure active or available data by requiring some authenticated access. This is also a good practice to prevent any changes to the data. At a minimum, you should encrypt historical archives. Any additional security measures, like signing, are an added benefit.

     

    Logs contain tons of useful information. When collected properly, they can help you improve security and quickly resolve network and systems issues. Creating a logging plan before you set up your log collection process can be the difference between long hours of digging through useless data and quickly finding what you need.

     

    Want more specific information? Check out these additional links:

    Log Management for a Security Pro

    Log Management for a Network Admin

    Log Management for a System Admin