cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

DevOps Pitfalls

Level 9

You've just been given the following assignment from your boss: He's asked you to make the organization more DevOps-oriented. You've gone back and forth on what his definition of DevOps is and what he wants you to accomplish. You're ready to get going, and think, "What could go wrong?" A good bit, it turns out. In the second post of this DevOps series, we are looking into some of the pitfalls and challenges of transitioning from a traditional operations model to a DevOps model.

Dev + DevOps + Ops

DevOps can help your organization when developers and operations are working together toward the same business goals. When you tighten communications, features and issues can be communicated back and forth fluidly between two teams.  Unfortunately, some organizations that have chosen to implement DevOps have placed a DevOps layer between developers and operations. Rather than increasing collaboration and communications between the developer and operations teams, this new team in between could potentially slow communications and the decision-making processes, resulting in failure. It's understandable to want people who are responsible for making this effort work, but rather than create a separate team, choose point people who will make sure that the development and operations teams are communicating effectively.

My Culture Just Ate Your DevOps

Saying that culture is an important piece in ensuring any projects' success is not a new concept. Having team buy-in on DevOps tasks is critical, and it's just as important to have a culture of communications and collaboration. But what if your organization's culture doesn't include that? Shaping an organization's culture, whether for DevOps or something else, can be a long and difficult process. There is not enough space in this post to talk about all of the things that could be done here.  Instead, start with addressing these three premises on how organizations can point their cultural ship in the right direction. First, as leadership, be the culture. You can't expect your organization to shift their cultural norms if the people at the top aren't exhibiting the culture that they want to see. The second suggestion to help you build your culture: Fake it 'til you make it. Lloyd Taylor, VP Infrastructure at Ngmoco said, "Behavior becomes culture." If you keep practicing the desired behaviors, eventually they become the culture. Finally, process becomes behavior. Having well-defined processes for how communications and DevOps interactions should work will build the behavior and in turn the culture.

Your Network Engineer Isn't a Developer

I use the term network engineer pretty loosely in the title of this section. For the purpose of my argument, it could be anyone on the operations side. Some organizations have many people in operations that do impressive scripting work to simplify tasks. These organizations may not have developers on staff and may look to these operators to be the developers in their DevOps initiatives. Don't get me wrong, for some organizations this may be exactly what they need and it may work out just fine. Unfortunately, what sometimes happens is the network engineer that is great at scripting lacks professional developer habits. Error handling, good software documentation, and regression testing are all things that an experienced developer would do out of habit that a network engineer might not think to address. (Okay, I'll admit that good software documentation can sometimes be difficult for experienced developers, too.) Can your operations person be a developer? They might have the skills and training, but don't count on them to be able to pull off all that a fully experienced developer can. If you happen to have an operations person that is a developer, maybe make that operator responsible for the points mentioned above.

Conversely, there are things on the operations side that your developers don't know about. For instance, your developers likely don't have deep knowledge of networking protocols, or even commands that would be required to configure them. One of the people that I spoke to for the last post experienced this very issue. In addition to experiencing long delivery times, his biggest concern was the appearance that the organization's developers were attempting to be network engineers when they were clearly not up for the job.

We Test in Production

It seems almost silly to mention this here, but you need to have a testing methodology for the product of your DevOps efforts. Even though it is basic, it's often overlooked that you shouldn't test your code in your production environment. If a large part of your DevOps program involves automation, and you roll out incorrect code, you roll out bad changes to your entire organization at crazy fast speeds. Yay, automation! Some development platforms allow you to test code to see what the output will be prior to running it in production. However, a solid test bed that replicates key components of the environment is helpful to have. The key is having an environment that you can quickly test and validate code behavior while not significantly slowing down the process for releasing the code to production.

Those are just a few of the pitfalls that frequently come up in a variety of DevOps initiatives. What do you think? What are the biggest challenges you've seen when someone is rolling out DevOps? Comment and let me know!

15 Comments
MVP
MVP

Nice write up

I still maintain that DevOps is nothing more than good customer relations when creating apps, writing software, designing a network, planning a database, etc.  Smoke & mirrors for a philosophy of how to do the job well and achieve a successful result.

Writing good code always required everything covered by DevOps.  At least, it did if you were successful at it--in EVERYONE'S eyes, not just yours, or just your teams.

It's always a team-building process (if you do it correctly), between the IT experts, their interdepartmental peers, their managers, and their clients.

MVP
MVP

Good information - combined with information from ThwackCamp 2017 DevOps takes on new approaches. Its really about perspective, but it also means learning new tools and new (dare I say better) ways of thinking.

Level 20

This is part of what scares me about devops... that changes are done in production.

MVP
MVP

Agreed, this used to be a joke, but now I guess it's reality.

Yes!

Buying bodies & licenses and hardware & time to have a Dev & Test & other sandboxed environments to prevent testing in Prod is expensive, but it prevents a lot of bad code from being exposed to clients/users.

While testing in Prod is the only "real" way to know if changes are good or not, doing so regularly seems like a career-limiting-decision.

Level 9

I think it boils down to cost now or cost later (when you potentially take down your environment and have to re-build it), but a lot of organizations won't see it like that.

Yes, there's too many ways to skin that cat.

Cost now comes with discovering limitations and perhaps needing to upgrade later for more cost.

Cost later means missing out on important labor-saving functionality today.

The litmus test is can you make more money (or reduce more trouble tickets / prevent problems / identify issues) by going with cost now, versus risking not dealing with those items today and losing time and customer satisfaction until you DO make the jump to the new products/environment?

MVP
MVP

I have a leader that calls them "Resume Generating Events."

Level 8

In some smaller businesses, the Network Admin wears many hats, to include: Network Admin, Windows / Server Admin, Developer, Applications specialist. I've been in many a small firm and have had to do multiple "jobs". Unfortunately, this will probably never change. On the good side, I've learned quite a bit and can "go with the flow" when needed.

There should be a large list of techy euphemisms for mistakes made at business.

It's one thing to "put your foot in your mouth" or "bark up the wrong tree", but it's quite another to participate in:

  • Resume Generating Events
  • Career Limiting Decisions
  • ?

This is why we have an automation team, not a dev-ops one. For us dev-ops is the agile company culture and a good robust pipeline to production.

"We test in PRD" was said by adatole​ during one of the Thwackcamp sessions and I nearly did a spit-take. I still can't get my head around it but I guess that won't matter. If the business has the stomach for it then so it shall be done.

I particularly enjoyed reading the section, "Your Network Engineer Isn't a Developer". If we are being honest with each other infrastructure is viewed as sandbags in the DevOps world. Always slowing progress down, being a speedbump/roadblock, getting in the way. For DevOps utopia infrastructure has to be limitless, without boundaries. No checks and balances...

Level 9

Testing in production just seems like setting yourself up for failure down the road if you're not extremely careful.  Sure some of the tests may work out fine, but no one is perfect in writing their code.  It's just a potential disaster I'd rather avoid.

Level 21

I have personally come to hate the term DevOps because it's so often mis-used that or it just isn't well defined, I am not sure which.  In my personal experience people seem to use it as more of a buzz word versus something with any actual meaning.  I have talked to several friends that also work in this industry and they have experienced similiar frustrations.  The problem with this is when the term is used but isn't well defined it leads to missed expectations.