THWACKcamp Preview: My Journey to DevOps Monitoring

narrow_bridge.jpg

It is true to the point of cliche that cloud came and changed everything, or at the very least is in the process of changing everything, even as we stand here looking at it. Workloads are moving to the cloud. Hybrid IT is a reality in almost every business. Terms and concepts like microservices, containers, and orchestration pepper almost every conversation in every IT department in every company.

Like many IT professionals, I wanted to increase my exposure without completely changing my career or having to carve out huge swaths of time to learn everything from the ground up. Luckily, there is a community ready-made to help folks of every stripe and background: DevOps. The DevOps mindset runs slightly counter to the traditional IT worldview. Along with a dedication to continuous delivery, automation, and process, a central DevOps tenet is, "It's about the people." This alone makes the community extremely open and welcoming to newcomers, especially newcomers like me who have an ops background and a healthy dose of curiosity.

So, for the last couple of years, I've been on a bit of journey. I wanted to see if there was a place in the DevOps community for an operations-minded monitoring warhorse like me. While I wasn't worried about being accepted into the DevOps community, I was worried if I would find a place where my interests and skills fit.

What concerned me the most was the use of words that sounded familiar but were presented in unfamiliar ways, chief among them the word "monitoring" itself. Every time I found a talk purporting to focus on monitoring, it was mostly about business analytics, developer-centric logging, and application tracing. I was presented with slides that presented such self-evident truths as:

"The easy and predictable issues are mostly solved for at scale, so all the interesting problems need high cardinality solutions."

Where, I wondered, were the hardcore systems monitoring experts? Where were the folks talking about leveraging what they learned in enterprise-class environments as they transitioned to the cloud?

I began to build an understanding that monitoring in DevOps was more than just a change of scale (like going from physical to virtual machines) or location (like going from on-premises to colo). But at the same time, it was less than an utterly different area of monitoring that had no bearing on what I've been doing for 20-odd years. What that meant was that, while I couldn't ignore DevOps' definition of monitoring, nor was I free to write it off as a variation of something I already knew.

Charity Majors (@mipsytipsy) has, for me at least, done the best job of painting a picture of what DevOps hopes to address with monitoring:

(excerpted from https://opensource.com/article/17/7/state-systems-administration)

"...And then on the client side: take mobile, for heaven's sake. The combinatorial explosion of (device types * firmwares * operating systems * apps) is a quantum leap in complexity on its own. Mix that in with distributed cache strategy, eventual consistency, datastores that split their brain between client and server, IoT, and the outsourcing of critical components to third-party vendors (which are effectively black boxes), and you start to see why we are all distributed systems engineers in the near and present future. Consider the prevailing trends in infrastructure: containers, schedulers, orchestrators. Microservices. Distributed data stores, polyglot persistence. Infrastructure is becoming ever more ephemeral and composable, loosely coupled over lossy networks. Components are shrinking in size while multiplying in count, by orders of magnitude in both directions... Compared to the old monoliths that we could manage using monitoring and automation, the new systems require new assumptions:

  • Distributed systems are never "up." They exist in a constant state of partially degraded service. Accept failure, design for resiliency, protect and shrink the critical path.
  • You can't hold the entire system in your head or reason about it; you will live or die by the thoroughness of your instrumentation and observability tooling.
  • You need robust service registration and discovery, load balancing, and backpressure between every combination of components..."

While Charity's post goes into greater detail about the challenge and some possible solutions, this excerpt should give you a good sense of the world she's addressing. With this type of insight, I began to form a better understanding of DevOps monitoring. But as my familiarity grew, so too did my desire to make the process easier for other "old monolith" (to use Charity's term) monitoring experts.

And this, more than anything else, was the driving force behind my desire to assemble some of the brightest minds in DevOps circles and discuss what it means, "When DevOps Says Monitor" for THWACKcamp this year. It is not too late to register for that session (https://thwack.solarwinds.com/community/thwackcamp-2017/when-devops-says-monitor). Better still, I hope you will join me for the full two-day conference and see what else might shake your assumptions, rattle your sense of normalcy, and set your feet on the road to the next stage of your IT journey.

PS: If you CAN'T join for the actual convention, not to worry! All the sessions will be available online to watch at a more convenient time.

Thwack - Symbolize TM, R, and C