Why Your Infrastructure Sucks For Automation

Having convinced myself in previous posts that my automation projects will never be finished and that I will forever be supporting 101 different device connection paradigms, I thought that in this post perhaps I should try to find some sliver of dry land in the fetid Swamp of Automation.


Failing that, let’s look at why consistency is so important when deploying infrastructure.

Your Infrastructure Sucks

It does. Don’t deny it. Despite best efforts, it’s still riddled with inconsistencies, plagued with one-off solutions, and besmirched with workarounds for badly-behaving applications.

Don’t feel bad; you are part of an unspoken global society held together by the silent bonds of engineering prowess. Our motto is “When every infrastructure is ‘special,’ every infrastructure is normal.

Deploy Consistently

Uniqueness, arguably, is the enemy of automation.

Effective automation requires consistency so the same encoded business logic and automation processes can be used across all devices. Exceptions and design variances mean that the generalized automation code has to be customized to deal with each unique situation, requiring further coding and creating additional complexity. This in turn increases both the cost and the time taken to developing automation tools.

This is one of the reasons why brownfield (swamp) automation deployments can be so difficult; years of inherited workarounds and one-off solutions can make even simple automation tasks seem impossibly complex because of the myriad “What if?” scenarios that have to be accounted for.

Greenfield solutions in contrast can provide fertile grounds for automation, but only if the infrastructure is designed with automation in mind from the start. A badly-designed greenfield infrastructure can be just as unmanageable as a brownfield.

If Proof Is Needed...

Amazon Web Services (AWS) for example—but in fact, almost any cloud provider—demonstrates these principles very effectively. When an application needs a feature or mechanism that AWS doesn’t offer, the choices are:

  • Rewrite the application; or
  • Don’t use AWS.

Because the AWS infrastructure is managed entirely by automation, there is no opportunity for one-off, non-standard solutions because the automation tools don’t support anything except the standard solutions. AWS is the concept of “one size fits all” taken to an absurd extreme. In fact, it’s almost risible how quickly programmers who previously would have had fits arguing that the infrastructure should do some incredible feats of engineering to support or otherwise mitigate lazy coding on their part, have adapted to the new paradigm where they have to code to meet the capabilities of the infrastructure. What is this crazy magic?

There is a lesson here, however. If vendors provide “nerd knobs” to allow us to engineer crazy solutions, and if we thrive on finding ways to make the infrastructure meet the needs of our users, then our users will expect us to keep using the nerd knobs and keep on coming up with insane solutions. And let’s be honest, as engineers, we’re kind of proud when we find a really clever way to accomplish what’s needed and we solve a problem. It goes against every problem-solving bone in our bodies to say “no.”

The difference is that in the cloud, the conversation goes like this:

 Programmer: “I need you to do this crazy thing so my app works”
      Cloud: “No.”
Programmer: “But don’t you have a nerd knob you can turn?”
      Cloud: “No.”
Programmer: “But without this, the project will fail.”
      Cloud: “Shame, but no.”
Programmer: “Seriously, you’re killing me here.”
      Cloud: “Don’t tempt me.”

The Sliver of Land

Imagine that the Swamp of Automation represents a brownfield automation deployment. It’s not that it can’t be done, but the inconsistencies can turn out to be lurking alligators with bad breath and worse tempers. Move slowly and tread carefully.

The sliver of buildable land in the middle of this swamp, if there is one, is a greenfield deployment where there’s an opportunity to build a deep foundation of automation before the first resident moves in. However, get the foundation wrong, and it may all yet sink into the ground.

Your Homework for the Holidays

Thinking about your current, sucky infrastructure, if you had the opportunity to build it from the ground up, what would you do differently? How would you make it easier to operate? Or do you already have amazing systems in place? If you’ve been involved in a greenfield deployment, was there any resistance to doing things in a way that would pay off down the road?

I’d love to hear your tales of competency, incompetency, triumph, and alligator bites.

  • Until there’s a compelling reason for the staff to embrace automation, it’ll flounder. Management loves automation because they don’t like surprises. Staff hate automation because it’s usually dependent on a programming language or scripting language that’s not common to the server admin’s skill set. Look at PowerShell. Remember when we were all going to learn how to PoSh our way to operations glory? Yeah, not so much. Solutions need problems. Automation is a solution to a problem many IT pros think doesn’t exist.

  • Does the "perfect" system or network even exists?
    All systems and networks are build, or tweaked, to serve a specific customer.
    I ounce bought a system of the shelve beacause we wanted something generic, 2 projects later we had such an unique system that even the original seller did not recognized it anymore.

    Since then i rely on what we've got and concider this as our perfect environment and don't worry about what others think, even wehn auditors say it is not as they would like to wsee it.
    My network runs for 6 years without problems or issues, well...writing this might awaken the gods of network interferance........, and we are happy with it.

  • Just started a new job where I'm tasked with looking at the evolved systems and trying to sort out the mess.  I'm being pulled in 20 different directions with no time to figure out the hows and whys of what was done in the past.  All this whilst not having high level access as "I'm new".  Might be interesting trying to pass my probation period.  At least there's a beer fridge in the office.

  • You are absolutely correct. My infrastructure sucks because of all the uniqueness, the general lack of adherence to process, and the constant change. Our infrastructure basically turns over every 3 years. I can't sell automation here because nothing stands still.

  • Watch out for the 'gators' and 'snakes' in the swamp and wear you waders 'cuz it's going to get deep.