cancel
Showing results for 
Search instead for 
Did you mean: 
mdriskell
Level 15

Re: Big Data or Big Garbage?

i spoke with two different engineers and posed the same question and both provided the same answer.  Honestly after we were told that we moved on and I didn't question it further.

0 Kudos
byrona
Level 21

Re: Big Data or Big Garbage?

Mrs. Y, I always find your discussion topics to be very interesting and thought provoking.

I completely agree that a lot of data being collected is "garbage data" and that data gluttony is a real problem which has me wondering what the driver for all of that is?  I don't think its generally the technical folks behind the problem, I think it's a combination of management and compliance requirements, or even a failure to understand compliance requirements.  The scenario I am thinking of goes something like this...  Management wants to meet some set of compliance requirements, reads a document on it and takes away that all data needs to be collected and kept for some large number of years and sends those requirements on to the technical folks.  While this scenario is a bit of an exaggeration, I have seen it more than once already.

I think that SIEM products provide a lot of value and there is a lot of value in log data so long as you scope things appropriately and have realistic expectations of the system, don't get caught up in the hype that many of these vendors want to sell you on.

byrona
Level 21

Re: Big Data or Big Garbage?

After you have had some time with Log Rhythm I would love to hear what you think of it.

0 Kudos
rharland2012
Level 15

Re: Big Data or Big Garbage?

The hype works very well to sell solutions....to the same management-level folks who actually sign the checks! This is why a good CIO is so valuable in shops large enough to warrant the hire. Stop the madness before it starts, normalize or boost expectations, and get the right tool for the job in the hands of those who do the job.

0 Kudos
jswan
Level 13

Re: Big Data or Big Garbage?

Mrs. Y, your problem is one of the reasons I continue to squirrel away plain text logs whenever possible... so when the tool de jour blows up I can go back to grep/awk/sed.

At the Bro Exchange last year this topic came up and I was fascinated to hear operators from one of the largest installations state that they had tried a huge laundry list of commercial and OSS log management tools before going back to a home-built tool that just runs grep in parallel across many machines.

0 Kudos
matt.matheus
Level 11

Re: Big Data or Big Garbage?

I think the difficulty is not in gathering enough useful information, but in finding the information you want in a reasonable amount of time.  When you are looking through 27 months of logs from a particular site, things that should come up as red alerts become buried in the chaff.  Knowledge on how to find what needs to be found / correlated... and the computing power to do it are in short supply for many organizations.  A good log management tool should be able to interpret logs and automatically parse for problems after being told what to look for.

Another significant problem is that the people conducting the investigations / reviewing logs aren't the ones writing the checks.  A salesperson can come in and talk at length about their whiz-bang product, throw in terms like compliance and visibility with a few buzzwords (like BIG DATA), and walk out with a purchase order.  Fast forward a few weeks when the product arrives and the people who actually will be needing to use it are forced to make a square peg fit into a round hole.

0 Kudos
Highlighted
mlan
Level 9

Re: Big Data or Big Garbage?

What do you think about Splunk?

0 Kudos
freid.42
Level 11

Re: Big Data or Big Garbage?

I really think the difficulty comes from the number of logs that are coming in from the number of devices. The more devices you add the more logging events that are going to be there. As the devices become more powerful and smarter the more they are going to put out. I feel that the number of logs coming from a bigger company can become unwieldy. This has become sure a burden for shops of any size.

We have started to use Symantec MSS for log evaluation. They collect logs from us and evaluate them and contact us for any issues that arise. This is convenient for the IT shop, but not everything is being logged. It's a cost vs risk kind of idea.

0 Kudos
IGFCSS.DSI
Level 10

Re: Big Data or Big Garbage?

Hi Mrs. Y,

I was reading your post and immediately thought of the book I'm currently reading on my Kindle, "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't" from Nate Silver. At a first glance (and even reading the first few pages) you can think it's about another subject but really it's about BIG DATA. Today we create some pentabytes of data every day and one of the major challenges we face is to filter the "signal" from the "noise". Or in Jerry Sto. Tomas words, filter the data from the garbage...

On the past forgotten times of "grep/sed/awk triple-threat" we really had to know what we were looking for otherwise those grep regex wouldn't work or returned us garbage. Somehow we passed from those "geek times" to a new generation of big data, fast data and click-click-click on a mouse button. Companies that produce/develop software, for example the analysis tools that you've mentioned, just deliver us that and leave it to us to have or not the mentioned PhD.

I'm remembering a famous software I've used a few years ago that is one of the best for SNMP monitoring and after install I've just added the IP address of a 48-port Layer 3 switch and "voilá"... Huge amount of graphs, counters, etc... And I though: "I don't need all of those graphs and counters. I don't want to graph bandwidth, in/out packets, in/out bytes for every switch port!!!". I've spent more time to delete the garbage than I've spend installing the darn thing.

Log analysis and correlation is even more difficult because our systems produce more and more information making it almost an "Hercules Task" for us to filter information.

Today when I need to implement such tools I spend more and more time to plan and design, then I implement the "sweet spot" or what I think it's my "sweet spot"...

0 Kudos
byrona
Level 21

Re: Big Data or Big Garbage?

Today when I need to implement such tools I spend more and more time to plan and design, then I implement the "sweet spot" or what I think it's my "sweet spot"...

Great point!  I think that there is often not enough time spent in the plan/design stage.  When it comes to SIEM products I think a lot of people go out and buy a product thinking "this will solve all of my log problems" based on the hype around the product without taking the time to identify the specific problems that you are trying to solve and then test the product to see if it will actually do as expected.  If you don't do this you end up with a solution in search of problem.

0 Kudos