cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

In Logging We Trust

Level 11

Happy - Bob Ross Meme.jpg

Everyone in IT loves to log. We love to log our servers, our networks, our security devices, and our security. We log all the things. Sometimes we even look at those logs, but mostly we dump them to a tool which paints a nice, happy little dashboard… just right there, and then we forget about it until we get those pesky notices that something has gone wrong.

The challenges here are myriad, however, and not always easy to address because they require political as well as technical fixes. IT personnel are generally great with technology, but not so much with the politicking. Kissing hands and shaking babies is apparently the wrong approach to take, and so when rebuffed by the suits, or sometimes the Bobs, we retreat to our happy little world of dashboards and log data.

One major challenge to logging is mentally getting beyond logging. We don’t need logging for loggings sake. What we actually need is correlation. What do I mean? I mean that all of the bits of information we collect from all of our disparate systems sit idly by, locked in their own little bubbles, sending occasional notices that send us squirreling off to solve a problem. None of the information we collect is analyzed collectively, it’s not correlated at all, and so we miss patterns.

Think about it. We collect from all of our systems in a structured way. We also collect vast amounts of machine data from systems as diverse as badge readers, BLE beacons, tweets, failed domain lookups, etc., but we don’t do anything with it as a whole. What we need to do is to start looking at our data instead of random datum. If we normalize as much of that data as possible, make intelligent connections between it all, and use intelligent analysis, we can start to make sense of the random noise on the wire and stop chasing squirrels.

The other problem we face is the major impediment to what I’ve just described: silos. Let’s face it, within the IT industry as a whole we segment ourselves off by specialty. Security, networking, systems, applications, storage, voice, wireless, and probably even more sub categories are all common areas of expertise, and those areas are very frequently operated as different departments within the larger IT organization, either de facto, or de jure. And as often as not those departments don’t work together, don’t always like each other, and sometimes even work against one another.

So, each segment of the IT organization is gathering data using different tools and methodologies, and with a varying amount of fidelity to what the data is telling them. Data correlation in the big picture is mostly worthless if it’s not done in a deliberate way across the entire organization. To get a detailed picture of your organization you need everything collected, not just the bits from groups who get along. Without that, you won’t realize the benefits of big data analytics, what I’ve been calling correlation, in any meaningful way. You won’t be able to connect the proverbial dots to a place from which valid, useful conclusions may be drawn. And without that, we might as well go back to our insular worlds, and work on our squirrel chasing.

7 Comments

SomeClown​, you make a good point.  Collecting logs for the sake of collecting is useless, what we do with them is where many organizations fall short.

MVP
MVP

It is a bit more than just correlation.  You need to set up filters for the "normal noise" and rules to catch the known issues.  Everything else then becomes suspect and needs to be looked at/categorized into

one of the 2 previous buckets.  This pares down the list on an ongoing basis.  Things become easier to spot that could be issues.  The term is management by exception.  It is a living thing that adjusts over time but allows you to get to the 'not normal or known' (exception) quicker.....

Level 9

We always need to sort and sift.  A way to see the grass through the weeds.  Each organization needs to address this according to their own needs.  However, collecting logs just to put a check in a check box, is a fruitless endeavor at best.

Level 14

Couldn't agree more.  We need to see the grass through the weeds.

Agreed on all points.  It's why SIEMs like Splunk and Sumo and Loggly and LogLogic were built.  Thank goodness--putting all the parts of the puzzle together--over and over and in new ways--is a job for a machine.  Whose output we must evaluate and take seriously and react to appropriately.

Level 10

Logging for the sake of logging is meaningless...what is important is correlation on the gathered logs to see entirely what these information are trying to tell us. I definitely assent to all the points mentioned above.

Level 20

Lol I remember that painter dude from PBS!

About the Author
Life-long and professional Network, VMware, and Unix Geek; Whiskey Taster; Brain Hacker; Student of Everything. Cancer Survivor. Armchair theoretical physicist.