cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

An Introduction to ELK Stack

Level 9

Wth the pace of business today, it’s easy to lose track of what’s going on. It’s also becoming increasingly difficult to derive value from data quickly enough that the data is still relevant. Oftentimes companies struggle with a situation where by the time the data has been crunched and visualized in a meaningful way, the optimal window for taking action has already come and gone.

One of the strategies that many organizations are using to make sense of the vast amounts of helpful data their infrastructure generates is by collecting all the logging information from various infrastructure components, crunching it by correlating time stamps and using heuristics that take relationships between infrastructure entities into account, and presenting it in a report or dashboard that brings the important metrics to the surface.

ELK Stack is one way modern organizations choose to accomplish this. As the name (“stack”) implies, ELK is not actually a tool in itself, but rather a useful combination of three different tools – Elasticsearch, Logstash, and Kibana – hence ELK. All three are open source projects maintained by Elastic. The descriptions of each tool from Elastic on their website are great, so I have opted not to re-write them. Elastic says they are:

  • Elasticsearch: A distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management. It combines the speed of search with the power of analytics via a sophisticated, developer-friendly query language covering structured, unstructured, and time-series data.
  • Logstash: A flexible, open source data collection, enrichment, and transportation pipeline. With connectors to common infrastructure for easy integration, Logstash is designed to efficiently process a growing list of log, event, and unstructured data sources for distribution into a variety of outputs, including Elasticsearch.
  • Kibana: An open source data visualization platform that allows you to interact with your data through stunning, powerful graphics. From histograms to geomaps, Kibana brings your data to life with visuals that can be combined into custom dashboards that help you share insights from your data far and wide.

Put simply, the tools respectively provide fast searching over a large data set, collect and distribute large amounts of log data, and visualize the collected and processed data. Getting started with ELK stack isn’t too difficult, but there are ways that community members have contributed their efforts to make it even easier. Friend of the IT community Larry Smith wrote a really helpful guide to deploying a highly available ELK stack environment that you can use to get going. Given a little bit of determination, you can use Larry’s guide to get a resilient ELK stack deployment running in your lab in an evening after work!

Alternatively, if you’re looking to get going on an enterprise-class deployment of these tools and don’t have time for fooling around, you could consider whether hosted ELK stack services would meet your needs. Depending on your budget and skills, it could make sense to let someone else do the heavy lifting, and that’s where services like Qbox come in. I’ve not used the service myself and I’m not necessarily endorsing this one, but I’ve seen manages services like this one be very successful in meeting other pressing needs in the past.

If you check this out and ELK Stack doesn’t meet your data insight requirements, there are other awesome options as well. There’s also the ongoing debate about proprietary vs. open source software and you’ll find that there are log collection/search/visualization tools for both sides of the matter. If you’re looking for something different, you may want to consider:

11 Comments
MVP
MVP

Very good to know there are open source equivalents to Splunk.

A challenge to many tools of this sort is that they are easy to get data into and build searches/queries to find things.  Getting meaningful results back out into a potentially actionable alert tends to be the challenge.  In other words, they don't share well with others once they get data in.

Level 17

Good insight, and options for running something other than Splunk..Thanks for the post!

ELK seems like just another flavor in an already saturated ice cream market, doing the same thing numerous other products do.

The most important information is missing: 

  • How does ELK compare to the other products?
  • What can ELK do better than the other product?
  • What's the mean time to resolution for problems, given it's Open Source?
  • What third-parties provide support for ELK?
  • What do third-party support vendors charge for helping resolve ELK issues, or for designing/implementing ELK in a custom environment?
Level 9

That important information might be missing if I had titled this post "Lessons Learned from Running ELK Stack In Production" or "Analytics Tools Showdown." 🙂

Alas, it's an introduction to what ELK Stack is.

I'm also not sure I agree with your conclusion that ELK is trying to enter a saturated market. I don't have stats to back this up, but I'm pretty sure it's tools like Elasticsearch that are the ones doing the saturating!

Cheers!

Level 12

As great and powerful as Splunk is, it costs all the money to get all the logs.  For smaller operations, this might be an interesting alternative.

Level 13

rschroeder​ has some great questions...custom and complex environments would be tricky without assistance.

MVP
MVP

Or for cheap organizations!  😉

Okay, so I had to Google ELK Stack as I had never heard of it before and I thought that this was a ruse. Who could blame me with names like: Elasticsearch, Logstash, and Kibana.

Nonetheless... interesting stuff.

Level 21

Organizations may think they are getting away with spending less $$$ but there are a lot of hidden costs in DIY and open source.

MVP
MVP

not to mention vulnerabilities and compatibility issues...

MVP
MVP

We have recently put in the elk stack , specifically with kirbana is there any integration with Solarwinds other than putting a html within a page?

About the Author
I grew up in Forest Lake, Minnesota in the 1960's, enjoying fishing, hunting, photography, bird watching, church, theater, music, mini-boggan, snowmobiling, neighborhood friends, and life in general. I've seen a bit, have had my eyes opened more than once, and tend not to make the same mistakes twice. Reinventing the wheel is not my preference, and if I can benefit from someone else's experience, that's good all the way around. If someone can benefit from my experience, it's why I share on Thwack.