Like many of you, I suspect, I am firmly under the thumb of my corporate security group. When they want to know what’s going on on the network I’ll put in as many taps as they want; but sometimes they want more than that, especially where NAT is involved and IPs and ports change from one side of a device to the other and it’s otherwise difficult to track sessions.
High Speed Logging
Have you heard of High Speed Logging (HSL)? I’d like to say that it’s a single standard that everybody plays by, but it seems like more of a concept that people choose to interpret in their own way. HSL is the idea of logging every single session, and in particular every single NAT translation.
So who offers HSL? Well, in theory if you can log session initiation to syslog, you can do High Speed Logging, but some vendors explicitly support this as a feature. For example:
- Cisco supports High Speed Logging for NAT translations in IOS XR, and Firewall HSL in IOS XE. Cisco uses a NetFlow-style logging mechanism.
- F5 LTM supports High Speed Logging for load balancer sessions, using a syslog-type message.
- Juniper’s Junos OS supports unstructured syslog, structured syslog and binary syslog (effectively structured syslog but better packed!).
Potential Road Blocks
High Speed Logging sounds simple enough, but there are a few things to consider before configuring it everywhere.
- What is the impact on your device when you enable High Speed Logging?
- Is it possible that the volume of logging messages generated might exceed the capacity of your device’s management port? One solution to this is to use an in-band destination for HSL instead.
- With multiple devices logging at high speed, can your management network cope with the aggregate throughput?
- Once you have the logs, how long do you have to keep those logs? Do you have the necessary storage available to you?
- For those logs to be useful, any system that is doing analysis, filtering or searching of the logs is going to have to be fairly fast to cope with the volume of data to look through.
- Will a DDoS attack also DoS your logging infrastructure?
The question of the whether a device is capable of supporting HSL should not be underestimated. In one company I worked at, the security team logged every session start and end on the firewalls. To accommodate this the firewall vendor had to provide a special logging-optimized code build, presumably at the cost of some throughput.
In another case I’ve seen a different firewall vendor’s control CPU knocked sideways just by asking it to count hits on every rule, which one would imagine should be way less intensive than logging every session.
Assuming you manage to get all those logs from device to log server, what are you going to do with them? What’s your interface to search this rapidly growing pile of log statements–undoubtedly in different formats–in a timely fashion? Are you analyzing real time and looking for threats? Do you pre-process all the logs into a common, searchable format, or do you only process when a search is initiated?
I suppose the broader question perhaps is whether High Speed Logging is actually the right solution to the problem of network visibility. Should we be doing something else using taps or SPAN ports instead? Having worked in an environment that logged every session on firewalls, I know it is tremendously useful when troubleshooting connectivity issues, but that doesn't make it the best or only solution.
What do you think? Have you tried High Speed Logging and succeeded? Or failed? Do you think your current log management products (or perhaps even just the hardware you run it on) are capable of coping with an onslaught of logs like that? What alternatives do you use to get visibility? Thanks in advance for sharing your thoughts!