Why Your Infrastructure Sucks For Automation

Having convinced myself in previous posts that my automation projects will never be finished and that I will forever be supporting 101 different device connection paradigms, I thought that in this post perhaps I should try to find some sliver of dry land in the fetid Swamp of Automation.


Failing that, let’s look at why consistency is so important when deploying infrastructure.

Your Infrastructure Sucks

It does. Don’t deny it. Despite best efforts, it’s still riddled with inconsistencies, plagued with one-off solutions, and besmirched with workarounds for badly-behaving applications.

Don’t feel bad; you are part of an unspoken global society held together by the silent bonds of engineering prowess. Our motto is “When every infrastructure is ‘special,’ every infrastructure is normal.

Deploy Consistently

Uniqueness, arguably, is the enemy of automation.

Effective automation requires consistency so the same encoded business logic and automation processes can be used across all devices. Exceptions and design variances mean that the generalized automation code has to be customized to deal with each unique situation, requiring further coding and creating additional complexity. This in turn increases both the cost and the time taken to developing automation tools.

This is one of the reasons why brownfield (swamp) automation deployments can be so difficult; years of inherited workarounds and one-off solutions can make even simple automation tasks seem impossibly complex because of the myriad “What if?” scenarios that have to be accounted for.

Greenfield solutions in contrast can provide fertile grounds for automation, but only if the infrastructure is designed with automation in mind from the start. A badly-designed greenfield infrastructure can be just as unmanageable as a brownfield.

If Proof Is Needed...

Amazon Web Services (AWS) for example—but in fact, almost any cloud provider—demonstrates these principles very effectively. When an application needs a feature or mechanism that AWS doesn’t offer, the choices are:

  • Rewrite the application; or
  • Don’t use AWS.

Because the AWS infrastructure is managed entirely by automation, there is no opportunity for one-off, non-standard solutions because the automation tools don’t support anything except the standard solutions. AWS is the concept of “one size fits all” taken to an absurd extreme. In fact, it’s almost risible how quickly programmers who previously would have had fits arguing that the infrastructure should do some incredible feats of engineering to support or otherwise mitigate lazy coding on their part, have adapted to the new paradigm where they have to code to meet the capabilities of the infrastructure. What is this crazy magic?

There is a lesson here, however. If vendors provide “nerd knobs” to allow us to engineer crazy solutions, and if we thrive on finding ways to make the infrastructure meet the needs of our users, then our users will expect us to keep using the nerd knobs and keep on coming up with insane solutions. And let’s be honest, as engineers, we’re kind of proud when we find a really clever way to accomplish what’s needed and we solve a problem. It goes against every problem-solving bone in our bodies to say “no.”

The difference is that in the cloud, the conversation goes like this:

 Programmer: “I need you to do this crazy thing so my app works”
      Cloud: “No.”
Programmer: “But don’t you have a nerd knob you can turn?”
      Cloud: “No.”
Programmer: “But without this, the project will fail.”
      Cloud: “Shame, but no.”
Programmer: “Seriously, you’re killing me here.”
      Cloud: “Don’t tempt me.”

The Sliver of Land

Imagine that the Swamp of Automation represents a brownfield automation deployment. It’s not that it can’t be done, but the inconsistencies can turn out to be lurking alligators with bad breath and worse tempers. Move slowly and tread carefully.

The sliver of buildable land in the middle of this swamp, if there is one, is a greenfield deployment where there’s an opportunity to build a deep foundation of automation before the first resident moves in. However, get the foundation wrong, and it may all yet sink into the ground.

Your Homework for the Holidays

Thinking about your current, sucky infrastructure, if you had the opportunity to build it from the ground up, what would you do differently? How would you make it easier to operate? Or do you already have amazing systems in place? If you’ve been involved in a greenfield deployment, was there any resistance to doing things in a way that would pay off down the road?

I’d love to hear your tales of competency, incompetency, triumph, and alligator bites.

  • Time - these things have built up over time, because of our own doing, sometimes due to the pressure of time and workload, sometimes because we took the quick-and-dirty solution and are paying for it now.

    Time, which is money as has been said, plus people's dedicated time, are what we need to fix things. The questions from that are:

    1. Can you afford it?

    2. Can you not afford it?

    One idea I've found to help with these situations is to not take vacation around the major holidays. The office empties out, demand for day-to-day work goes down, and I can sit and focus for long times, broken only by the need for coffee or lunch or... and I am able to design and code automated solutions. Just this week I've finished rewriting an old, partially manual process that was running on a legacy system. Now it will be an automated process! (At least until someone decides to replace this new system with a shiny new toy.)

  • I push every engineer that walks through the door to build what is needed for today...do not just tack on to the existing network.  I say "How would you build this if it was new?" and "Would you have built it this way?"


  • Glad everyone is in the same boat. Whenever I do consulting, there is only a bit of automation I can implement. Many things are unfortunately a manual process.  

  • I'm really concerned... now they're talking about moving to O365... I'm not excited about it at all.  The only good thing is the mailbox size I think.

  • Yes, this describes our network environment.  Required to support all things from all vendors instead of requiring consistency.

    Let me know when there's a way out.

    Move ALL things to AWS?  Not an option yet.  And it can't be an option until "we" (the purchasers) stop buying products that are one-offs, that don't comply with standards and best practices.