Showing results for 
Search instead for 
Did you mean: 
Create Post

Why Your Infrastructure Sucks For Automation

Level 13

Having convinced myself in previous posts that my automation projects will never be finished and that I will forever be supporting 101 different device connection paradigms, I thought that in this post perhaps I should try to find some sliver of dry land in the fetid Swamp of Automation.


Failing that, let’s look at why consistency is so important when deploying infrastructure.

Your Infrastructure Sucks

It does. Don’t deny it. Despite best efforts, it’s still riddled with inconsistencies, plagued with one-off solutions, and besmirched with workarounds for badly-behaving applications.

Don’t feel bad; you are part of an unspoken global society held together by the silent bonds of engineering prowess. Our motto is “When every infrastructure is ‘special,’ every infrastructure is normal.

Deploy Consistently

Uniqueness, arguably, is the enemy of automation.

Effective automation requires consistency so the same encoded business logic and automation processes can be used across all devices. Exceptions and design variances mean that the generalized automation code has to be customized to deal with each unique situation, requiring further coding and creating additional complexity. This in turn increases both the cost and the time taken to developing automation tools.

This is one of the reasons why brownfield (swamp) automation deployments can be so difficult; years of inherited workarounds and one-off solutions can make even simple automation tasks seem impossibly complex because of the myriad “What if?” scenarios that have to be accounted for.

Greenfield solutions in contrast can provide fertile grounds for automation, but only if the infrastructure is designed with automation in mind from the start. A badly-designed greenfield infrastructure can be just as unmanageable as a brownfield.

If Proof Is Needed...

Amazon Web Services (AWS) for example—but in fact, almost any cloud provider—demonstrates these principles very effectively. When an application needs a feature or mechanism that AWS doesn’t offer, the choices are:

  • Rewrite the application; or
  • Don’t use AWS.

Because the AWS infrastructure is managed entirely by automation, there is no opportunity for one-off, non-standard solutions because the automation tools don’t support anything except the standard solutions. AWS is the concept of “one size fits all” taken to an absurd extreme. In fact, it’s almost risible how quickly programmers who previously would have had fits arguing that the infrastructure should do some incredible feats of engineering to support or otherwise mitigate lazy coding on their part, have adapted to the new paradigm where they have to code to meet the capabilities of the infrastructure. What is this crazy magic?

There is a lesson here, however. If vendors provide “nerd knobs” to allow us to engineer crazy solutions, and if we thrive on finding ways to make the infrastructure meet the needs of our users, then our users will expect us to keep using the nerd knobs and keep on coming up with insane solutions. And let’s be honest, as engineers, we’re kind of proud when we find a really clever way to accomplish what’s needed and we solve a problem. It goes against every problem-solving bone in our bodies to say “no.”

The difference is that in the cloud, the conversation goes like this:

 Programmer: “I need you to do this crazy thing so my app works”
      Cloud: “No.”
Programmer: “But don’t you have a nerd knob you can turn?”
      Cloud: “No.”
Programmer: “But without this, the project will fail.”
      Cloud: “Shame, but no.”
Programmer: “Seriously, you’re killing me here.”
      Cloud: “Don’t tempt me.”

The Sliver of Land

Imagine that the Swamp of Automation represents a brownfield automation deployment. It’s not that it can’t be done, but the inconsistencies can turn out to be lurking alligators with bad breath and worse tempers. Move slowly and tread carefully.

The sliver of buildable land in the middle of this swamp, if there is one, is a greenfield deployment where there’s an opportunity to build a deep foundation of automation before the first resident moves in. However, get the foundation wrong, and it may all yet sink into the ground.

Your Homework for the Holidays

Thinking about your current, sucky infrastructure, if you had the opportunity to build it from the ground up, what would you do differently? How would you make it easier to operate? Or do you already have amazing systems in place? If you’ve been involved in a greenfield deployment, was there any resistance to doing things in a way that would pay off down the road?

I’d love to hear your tales of competency, incompetency, triumph, and alligator bites.

Level 13

I've always been a great advocate of "Keep it Simple" the more complicated you make it the more difficult it is to troubleshoot and fix later. Nothing is ever perfect but we don't need to make things difficult for ourselves.

Level 14

Thanks for the article.  We've not yet fully dove into automation, due to exactly what you said.........uniqueness.  This is truly a can of worms that can have a lot of hurdles. 

Level 11

It's not the problem that "our" infrastructure sucks, it's more the legacy applications that are being run on said infrastructure that cause things to suck.  Companies are often to reliant upon legacy programs that can only run on xyz, and they're unwilling to invest in a solution as "It works now, so why change it".

Automation is great, and in time will put most of us out of a job, whether it be through ACI, or Ansible or Cloud, you're always going to need people to support "legacy" kit as companies are just not ready to turn the lights off just yet.  And the more things like the TSB meltdown that happen, then the longer automation is going to take to cement itself as the future.

Level 13

The funny thing is that when you push those apps into the cloud, it seems that quite often resources are magically found to update the applications to make them work (whereas while it was being propped up by the infrastructure internal to the company, nobody cared).

Our current Data Center needs to be torn down in 2 years to make room for other stuff. Our new Data Center 10 feet away will ideally be ready in a year to start installing new stuff. We are keeping our IP's and streamlining the move as much as possible by stretching the network from the old to the new. While it seems like alot will on the surface stay the same, everything that gets us there could change. Currently we are in a bake off for network technologies, Cisco versus Arista. They are being evaluated both traditionally and in an SDN configuration. We also are looking at NSX from VMWare at that layer. Things like the hardware that runs our VMWare infrastructure might change too. Recently we have been a Cisco UCS and Pure Storage shop, but much of what will go into the the new data center will be net new. Everything is on the table, and we are looking to standardize as much as possible.

Level 9

Nice to know we are normal.

Yes, this describes our network environment.  Required to support all things from all vendors instead of requiring consistency.

Let me know when there's a way out.

Move ALL things to AWS?  Not an option yet.  And it can't be an option until "we" (the purchasers) stop buying products that are one-offs, that don't comply with standards and best practices.


I'm really concerned... now they're talking about moving to O365... I'm not excited about it at all.  The only good thing is the mailbox size I think.

Glad everyone is in the same boat. Whenever I do consulting, there is only a bit of automation I can implement. Many things are unfortunately a manual process.  

I push every engineer that walks through the door to build what is needed for not just tack on to the existing network.  I say "How would you build this if it was new?" and "Would you have built it this way?"


Level 12

Time - these things have built up over time, because of our own doing, sometimes due to the pressure of time and workload, sometimes because we took the quick-and-dirty solution and are paying for it now.

Time, which is money as has been said, plus people's dedicated time, are what we need to fix things. The questions from that are:

1. Can you afford it?

2. Can you not afford it?

One idea I've found to help with these situations is to not take vacation around the major holidays. The office empties out, demand for day-to-day work goes down, and I can sit and focus for long times, broken only by the need for coffee or lunch or... and I am able to design and code automated solutions. Just this week I've finished rewriting an old, partially manual process that was running on a legacy system. Now it will be an automated process! (At least until someone decides to replace this new system with a shiny new toy.)

Level 16

Watch out for the 'gators' and 'snakes' in the swamp and wear you waders 'cuz it's going to get deep.

You are absolutely correct. My infrastructure sucks because of all the uniqueness, the general lack of adherence to process, and the constant change. Our infrastructure basically turns over every 3 years. I can't sell automation here because nothing stands still.

Level 14

Just started a new job where I'm tasked with looking at the evolved systems and trying to sort out the mess.  I'm being pulled in 20 different directions with no time to figure out the hows and whys of what was done in the past.  All this whilst not having high level access as "I'm new".  Might be interesting trying to pass my probation period.  At least there's a beer fridge in the office.

Level 8

Does the "perfect" system or network even exists?
All systems and networks are build, or tweaked, to serve a specific customer.
I ounce bought a system of the shelve beacause we wanted something generic, 2 projects later we had such an unique system that even the original seller did not recognized it anymore.

Since then i rely on what we've got and concider this as our perfect environment and don't worry about what others think, even wehn auditors say it is not as they would like to wsee it.
My network runs for 6 years without problems or issues, well...writing this might awaken the gods of network interferance........, and we are happy with it.


Until there’s a compelling reason for the staff to embrace automation, it’ll flounder. Management loves automation because they don’t like surprises. Staff hate automation because it’s usually dependent on a programming language or scripting language that’s not common to the server admin’s skill set. Look at PowerShell. Remember when we were all going to learn how to PoSh our way to operations glory? Yeah, not so much. Solutions need problems. Automation is a solution to a problem many IT pros think doesn’t exist.