cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

THWACKcamp Preview: My Journey to DevOps Monitoring

Level 18

narrow_bridge.jpg

It is true to the point of cliche that cloud came and changed everything, or at the very least is in the process of changing everything, even as we stand here looking at it. Workloads are moving to the cloud. Hybrid IT is a reality in almost every business. Terms and concepts like microservices, containers, and orchestration pepper almost every conversation in every IT department in every company.

Like many IT professionals, I wanted to increase my exposure without completely changing my career or having to carve out huge swaths of time to learn everything from the ground up. Luckily, there is a community ready-made to help folks of every stripe and background: DevOps. The DevOps mindset runs slightly counter to the traditional IT worldview. Along with a dedication to continuous delivery, automation, and process, a central DevOps tenet is, "It's about the people." This alone makes the community extremely open and welcoming to newcomers, especially newcomers like me who have an ops background and a healthy dose of curiosity.

So, for the last couple of years, I've been on a bit of journey. I wanted to see if there was a place in the DevOps community for an operations-minded monitoring warhorse like me. While I wasn't worried about being accepted into the DevOps community, I was worried if I would find a place where my interests and skills fit.

What concerned me the most was the use of words that sounded familiar but were presented in unfamiliar ways, chief among them the word "monitoring" itself. Every time I found a talk purporting to focus on monitoring, it was mostly about business analytics, developer-centric logging, and application tracing. I was presented with slides that presented such self-evident truths as:

"The easy and predictable issues are mostly solved for at scale, so all the interesting problems need high cardinality solutions."

Where, I wondered, were the hardcore systems monitoring experts? Where were the folks talking about leveraging what they learned in enterprise-class environments as they transitioned to the cloud?

I began to build an understanding that monitoring in DevOps was more than just a change of scale (like going from physical to virtual machines) or location (like going from on-premises to colo). But at the same time, it was less than an utterly different area of monitoring that had no bearing on what I've been doing for 20-odd years. What that meant was that, while I couldn't ignore DevOps' definition of monitoring, nor was I free to write it off as a variation of something I already knew.

Charity Majors (@mipsytipsy) has, for me at least, done the best job of painting a picture of what DevOps hopes to address with monitoring:

(excerpted from https://opensource.com/article/17/7/state-systems-administration)

"...And then on the client side: take mobile, for heaven's sake. The combinatorial explosion of (device types * firmwares * operating systems * apps) is a quantum leap in complexity on its own. Mix that in with distributed cache strategy, eventual consistency, datastores that split their brain between client and server, IoT, and the outsourcing of critical components to third-party vendors (which are effectively black boxes), and you start to see why we are all distributed systems engineers in the near and present future. Consider the prevailing trends in infrastructure: containers, schedulers, orchestrators. Microservices. Distributed data stores, polyglot persistence. Infrastructure is becoming ever more ephemeral and composable, loosely coupled over lossy networks. Components are shrinking in size while multiplying in count, by orders of magnitude in both directions... Compared to the old monoliths that we could manage using monitoring and automation, the new systems require new assumptions:

  • Distributed systems are never "up." They exist in a constant state of partially degraded service. Accept failure, design for resiliency, protect and shrink the critical path.
  • You can't hold the entire system in your head or reason about it; you will live or die by the thoroughness of your instrumentation and observability tooling.
  • You need robust service registration and discovery, load balancing, and backpressure between every combination of components..."

While Charity's post goes into greater detail about the challenge and some possible solutions, this excerpt should give you a good sense of the world she's addressing. With this type of insight, I began to form a better understanding of DevOps monitoring. But as my familiarity grew, so too did my desire to make the process easier for other "old monolith" (to use Charity's term) monitoring experts.

And this, more than anything else, was the driving force behind my desire to assemble some of the brightest minds in DevOps circles and discuss what it means, "When DevOps Says Monitor" for THWACKcamp this year. It is not too late to register for that session (https://thwack.solarwinds.com/community/thwackcamp-2017/when-devops-says-monitor). Better still, I hope you will join me for the full two-day conference and see what else might shake your assumptions, rattle your sense of normalcy, and set your feet on the road to the next stage of your IT journey.

PS: If you CAN'T join for the actual convention, not to worry! All the sessions will be available online to watch at a more convenient time.

6 Comments

I felt the same way when Security / Cisco came through and said we needed to do "segmentation" of networks.  "Yep, already on top of that" I thought to myself.  "That's old news, history we already took care of" I kept thinking.

Then I realized they meant "You need to identify every necessary packet flow and have an ACL customized to the switch ports it crosses, and turn 100,000+ switchports into 100,00+ firewalls by deploying ISE for your NAC."

Uh . . .   OK . . .?

Which precipitated months of testing & trial & error and the discovery that Cisco's 2960S switch stacks don't have the memory/CPU resources to deploy ISE, necessitating the replacement of ~400 48-port switches in my network.

Sigh . . .

New terms, new ways of doing things, new products, new ways to get money from one budget into Cisco's or my VAR's . . .

And this is just a tiny under-the-rugs example of DevOps monitoring.  Not like a B.O.A.T., where ownership of a fine pleasure craft (even if only a little outboard motorboat) means "Break Out Another Thousand" for some folks.  Maybe "Break Out Another Million" should be the new DevOps / Security acronym?

The plot thickens, and I'll be really interested to hear how SWI is planning to tool up to slay this new, complex, monitoring Dragon!

MVP
MVP

Level 13

Count down has started

Level 21

Unfortunately I didn't attend this track at Thwack camp so I am going to have to go back and watch this one.

MVP
MVP

This meshes with at least one session of this years camp and its all about re-thinking how and why we do things.

About the Author
In my sordid career, I have been an actor, bug exterminator and wild-animal remover (nothing crazy like pumas or wildebeasts. Just skunks and raccoons.), electrician, carpenter, stage-combat instructor, American Sign Language interpreter, and Sunday school teacher. Oh, and I work with computers. Since 1989 (when you got a free copy of Windows 286 on twelve 5¼” floppies when you bought a copy of Excel 1.0) I have worked as a classroom instructor, courseware designer, desktop support tech, server support engineer, and software distribution expert. Then about 14 years ago I got involved with systems monitoring. I've worked with a wide range of tools: Tivoli, Nagios, Patrol, ZenOss, OpenView, SiteScope, and of course SolarWinds. I've designed solutions for companies that were extremely modest (~10 systems) to those that were mind-bogglingly large (250,000 systems in 5,000 locations). During that time, I've had to chance to learn about monitoring all types of systems – routers, switches, load-balancers, and SAN fabric as well as windows, linux, and unix servers running on physical and virtual platforms.