cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Automation Paralysis: Why We Get Stuck Automating The Small Stuff

Level 13

Despite all the talk of all our jobs being replaced by automation, my experience is that the majority of enterprises are still very much employing engineers to design, build, and operate the infrastructure. Why, then, are most of us stuck in a position where despite experimenting with infrastructure automation, we have only managed to build tools to take over small, mission-specific tasks, and we've not achieved the promised nirvana of Click Once To Deploy?

We Are Not Stupid

Before digging into some reasons we're in this situation, it's important to first address the elephant in the room, which is the idea that nobody is stupid enough to automate themselves out of a job. Uhh, sure we are. We're geeks, we're driven by an obsession with technology, and the vast majority of us suffer from a terrible case of Computer Disease. I believe that the technical satisfaction of managing to successfully automate an entire process is a far stronger short-term motivation than any fear of the potential long-term consequences of doing so. In the same way that hoarding information as a form of job security is a self-defeating action (as Greg Ferro correctly says, "if you can't be replaced, you can't be promoted"), avoiding automation because it takes a task away from the meatbags is an equally silly idea.

Time Is Money

Why do we want to automate? Well, automation is the path to money. Automation leads to time-saving; time-saving leads to agility; agility leads to money. Save time, you must! Every trivial task that can be accomplished by automation frees up time for more important things. Let's be honest, we all have a huge backlog of things we've been meaning to do, but don't have time to get to.

However, building automation takes time too. There can be many nuances to even simple tasks, and codifying those nuances and handling exceptions can be a significant effort, and large scale automation is exponentially more complex. Because of that, we start small, and try to automate small steps within the larger task because that's a manageable project. Soon enough, there will be a collection of small, automated tasks built up, each of which requires its own inputs and generates its own outputs, and--usually--none of which can talk to each other because each element was written independently. Even so, this is not a bad approach, because if the tasks chosen for automation occur frequently, the time saved by the automation can outweigh the time spent developing it.

This kind of automation still needs hand-holding and guidance from a human, so while the humans are now more productive, they haven't replaced themselves yet.

Resource Crunch

There's an oft-cited problem that infrastructure engineers don't understand programming and programmers don't understand infrastructure, and there's more than a grain of truth to this idea. Automating small tasks is something that many infrastructure engineers will be capable of, courtesy of some great module/package support in scripting languages like Python. Automating big tasks end-to-end is a different ball game, and typically requires a level of planning and structure in the code exceeding that which most infrastructure engineers have in their skills portfolio. That's to be expected: if coding was an engineer's primary skill, they'd more likely be a programmer, not an infrastructure engineer.

Ultimately, scaling an automation project will almost always require dedicated and skilled programmers, who are not usually found in the existing team, and that means spending money on those programming resources, potentially for an extended period of time. While the project is running, it's likely that there will be little to no return on the investment. This is a classic demonstration of the maxim that you have to speculate to accumulate, but many companies are not in a position--or are simply unwilling--to invest that money up front.

The Cliffs Of Despair

With this in mind, in my opinion, one of the reasons companies get stuck with lots of small automation is that it's relatively easy to automate multiple, small tasks, but taking the next step and automating a full end-to-end process is a step too far for many companies. It's simply too great a conceptual and/or financial leap from where things are today. Automating every task is somewhere so far off in the distance, nobody can even forecast it.

They say a picture is worth a thousand words, which probably means I should have just posted this chart and said "Discuss," but nonetheless, as a huge fan of analyst firms, I thought that I could really drive my point home by creating a top quality chart representing the ups and downs of infrastructure automation.

cliffs_of_despair.png

As is clearly illustrated here, after the Initial Learning Pains, we fall into the Trough Of Small Successes, where there's enough knowledge now to create many, small automation tools. However, the Cliffs Of Despair loom ahead as it becomes necessary to integrate these tools together and orchestrate larger flows. Finally–and after much effort–a mechanism emerges by which the automation flows can be integrated, and the automation project enters the Plateau of Near Completion where the new mechanism is applied to the many smaller tools and good progress is made towards the end goal of full automation. However, just as the project manager announces that there are only a few remaining tasks before the project can be considered a wrap, development enters the Asymptotic Dream Of Full Automation, whereby no matter how close the team gets to achieving full automation, there's always just one more feature to include, one more edge case that hadn't arisen before, or one more device OS update which breaks the existing automation, thereby ensuring that the programming team has a job for life and will never achieve the sweet satisfaction of knowing that the job is finished.

Single Threaded Operation

There's one more problem to consider. Within the overall infrastructure, each resource area (e.g., compute, storage, network, security) is likely working their own way towards the Asymptotic Dream Of Full Automation and at some point will discover that full, end-to-end automation means orchestrating tasks between teams. And that's a whole new discussion, perhaps for a later post.

Change My Mind

change_my_mind.png

26 Comments
Level 14

Great article!  I thoroughly enjoyed this one. 

Level 13

good Article. Thanks

I definitely detect a taste of The Princess Bride in your graph.

Level 20

I completely agree... we don't need to stinkin' automate things for the sake of automating things.

Level 16

Great graph!

Having just realized your chart reveals what you said (that the automation goal will never be achieved) I'm falling victim to my tendency to explore alternative answers.  OK, call it playing Devil's Advocate, or being just plain challenging.

I want to view this from the other side of the graph--from those end points that may never touch, and work backwards to see what has to happen to make those lines meet at the end.

Does it take redefining what "automation" means?  Or evaluating what examples we can find that show successful automation?

Perhaps automating those little things, and adding them together, make the big things happen.

Yes, ridiculous amounts of work and planning and coordination and trial & error are involved.  But I'm betting Amazon didn't start out fully formed.  Nor did any computerized and automated integrated circuit soldering robot.

You hit it on the head when you talked about how combining the smaller pieces into a greater sum.

Now if I could only get the top folks to include the Network team in the design of every new building BEFORE someone has already laid claim to all of the floor space, I could show them how to reduce infrastructure costs by better planning and layout of the network rooms and their location within a building.

Level 13

Thanks for the post.  Really enjoying this thread.

Level 14

We never get time to do more than the basic automation.  Even then we have to scrounge the time to do it from somewhere else.  The time saved never seems to materialise as something else will always crop up to fill the gap.

MVP
MVP

Good article

MVP
MVP

Gotta say I love the meme. I get the reference!!!!

Level 13

I ran the graph past my wife for her comments, and she asked if I thought it would work. My conclusion was that it would take a miracle... 😕

Level 13

Part of the problem is the whole "boil the ocean" paradigm. And during those ridiculous amounts of work and planning and coordination, nothing is working and there's a danger that support for the initiative wanes. To your point about getting in at the planning layer, that's exactly it; if you build something out from the start with an end goal (automation in this case) in mind, perhaps it gets built differently than the infrastructure (and organization) we actually have today. Many of us are in the unfortunate position of trying to automate in hindsight, which can be extremely challenging.

Level 13

That's another argument put forward for why we won't automate ourselves out of a job; there's always something else you can do with the time you free up!

Level 13

*grin* Thank you for saying so (I feel somewhat vindicated now!)

Fantastic article! Got the Pete Monaghan 5-star approval. I loved the graph too. Similar to the graph we use for SAP implementation projects. The teams I work with here are challenged with the effort to automate small and mundane tasks.

And no, I won't change your mind... because I agree with you.  🙂

So you and rschroeder bring bring up an interesting conundrum, jgherbert​...a proverbial "chicken and egg" scenario, if you will.  If one "Begins with the end in mind", as Stephen Covey would have us do, and the end is automation, what happens when dead ends, pitfalls, cliffs, etc. crop up along the way...DevOps and SDLC (yep, I'm an old dawg!) be dammed [sic] in this case (sorry, GEEKS!)?  How do we keep up the momentum (or CAN we?)  Conversely, if one uses good ole "Reverse Engineering" and works their way back from, what did you call it, "Promised Nirvana", one might not get to the starting line because, as we see, something crops up in the middle that presents us with an uncrossable chasm:

Image result for then a miracle occurs cartoon

I guess this can happen either way but I absolutely appreciate where you are going with this and, like you I think, my philosophy falls in the "work smarter, not harder" camp.  IMHO, the only reason anyone would worry about "automating themselves out of a job" is that they are more focused on themselves than the organization (sorry, harsh I know but the truth does sting sometimes).  When we take the focus off of ourselves and put it on others in the organization, we give others a chance to learn, grow and shine.  If not, we become silos and blockages to growth.  Like Peter Diamandis stated, "Bureaucracy is an obstacle to be conquered with persistence, confidence and a bulldozer when necessary."

As for how we get through the valleys and cliffs, this is where good mission/vision/values statements come into play.  At some level, I think, we all want to be part of something greater than ourselves  If we keep an eye on strategy (long-term) and focus on tactics (short-term), we stand a greater chance of making the rough places plain and not having to try and drive by looking in the rear-view mirror.  And remember:

Meat Loaf - Objects in the Rear View Mirror May Appear Closer They Are - YouTube

I like your thoughts, and I'll only submit one thought about NOT being able to build with the end in mind:

I can design and build and provide a network based on Management's guidelines for what they want it to do at Date X. 

Management is unwilling to provide those specifics.  They expect my team to see the future and provide a fast and resilient and secure network for all possible outcomes.

We know how that works out.  Some new technology is released and is purchased by an internal splinter group.  It's incompatible with the network or our security protocols or policies, but we are required to  accommodate it because the capital has been spent to purchase it.

It's Management's "fault" for not requiring purchases to be vetted by IT and Security before releasing the funds.  And they are also culpable for allowing an environment to exist where end users and department heads aren't trained / required to come to IT and Security with new ideas BEFORE those users approach vendors with a perceived need.

In an ideal world we might also magically prohibit users from being approached by salespeople who are offering poor or incompatible products.  But that's a magic wand I can't have or wave.

I'd settle for someone drawing a line in the sand and saying "In 24 months we want the network to look like this, perform in such a manner, pass these tests and become certified to standards X, Y, and Z".

I could design it, get quotes, submit them for approval, and begin building or compromising (depending on funding).  I could also teach Management why some things they ask for are expensive or impossible--or completely practical and easily accomplished.

Poke me when we get there so I can wake up in "Rick Perfect World."

Level 14

Are you sure you don't work here.  It all sounds very very familiar.  We have management who tell us techies how to actually do our job without actually understanding what they are talking about. I keep telling them to tell me what they want to achieve and I will come up with some solutions for them to choose between.  I will then implement the chosen solution.  Don't even get me started on shadow IT.  Our HR system was put in without ITs knowledge.  Every time there is a problem they log a ticket with us saying it is an ADFS or AD issue.  It doesn't use ADFS or AD and we don't support it.  Management still expect us to fix it.

Level 14

It's OK putting the organisation first but that doesn't pay the mortgage or put food on the table.  Yes we should do our best for the organisation that pays us but we still need to look out for ourselves and our families.  Keeping an eye on the rear view mirror is a good thing but we should also be looking forward to where we are going.

You are dead on, Rick, and I agree with everything you said.

Because we exist in such a fluid and dynamic world these days, it is pretty near impossible to end up EXACTLY where you started out to get.  Even making minor course corrections, like Apollo 11 did, we are still "off course" 90+% of the time.  I believe that all we as IT pros can do is our best to anticipate the needs of the organization, while remaining flexible and rigid (yes, there, I said it!) all at the same time.

I remember at my former job, there was a development manager who wanted to create an environment where people could come in and be, well, "creative".  He wanted a place where people could come and generate fresh, new ideas for products (the company was struggling at the time, but that's another story).  So they took a larger conference room and transformed it: corrugated tin on the walls, blackboard and whiteboard material everywhere, big screen 32" (OK, this was a few years ago!) TVs, etc.  And I recall what my manager said to me about the IT needs: Anything this manager wants, he gets.  No arguments, excuses, or exceptions.  I would put quotes around it but there's been a few minutes between there and here, but that was pretty much it.  Whatever, whenever, however, didn't matter.  We had to do it.  I still remember to this day the "nickname" that all of us who were in IT at the time gave this shiny penny: "The Romper Room"!

As I have said before, IT would be great if it wasn't for all the users.  I can't tell you how many times I've gone around to sites and discovered dumb switches and hubs.  And the trouble is, those dumb devices are connected to mission critical systems.  Makes me cringe.  I then have to go to management and say "What the <BLEEP> is going on here?  This isn't how we do things!".  Then we have to take the site down and swap out the dumb equipment for Cisco stuff, which takes time and money.

So, yeah, the moment you wake up in your "Rick Perfect World", please let me know so I can wake up in mine! 😉

Level 13

Absolutely true. I thought I might have some spare time a few years ago, but I was mistaken.  It never ends. 

Level 12

Great, now I want an MLT.

It's ironic, isn't it petergwilson​, how so much of the "IT experience" is universal.  That's why "Dilbert" was - and IS - so very popular.

I remember having a boss when I first started in the "professional" computer arena (read "sales"), and he was EXACTLY like the "Big Boss" in "Dilbert".  There was a "Dilbert" animated show on way, way back and I just remember the scene where the "Big Boss" blusters into the conference room saying, "Am I late? Am I late?!!"  Wally says, "No, you're thirty seconds early." and the Big Boss says, "Oh, good! I have time to make a phone call!"  That was my boss to a tee!  I would be sitting in my weekly one-on-one with him, the phone would ring and he would take the call.  I look back now and chuckle at him, but at the time it was a little hurtful.  But hey, then as now; I've got big shoulders.

Point being that we are not alone in this world of IT!  That's why I love this community of THWACK: I can come here, hang out for a little while and feel encouraged by others who talk about, and are working through, their difficulties.  If I can, in turn, encourage just one other person to "hang in there" and work through their struggles, then my time here will have been worth it.

Great group and great topics!

Too true, PGW!  Again, it's that delicate balance of work and home life.  I have seen several folks in another thread saying that they don't want to do anyting in this forum outside of "work", and I admire that.  It also comes down to how badly do you (pl) want what you say you want.  If you want something badly enough, you will figure out a way of making it happen, even if that displaces the "balance" for a short time.  And, actually, looking out for the company DOES pay the mortgage because in doing so, the company stays in business. And if the company goes out of business, we have no job to pay the mortgage.  Just food for thought...

Level 9

Great article, robots cannot do all what humans can.

Level 13

Going back and re-reading these after the latest post.  Couldn't agree more.  Great series.