How do you define DevOps and what are the main benefits of adopting it in your organization?

So I spent a couple years at a very large enterprise as they went into their devops and modernization journey.  There the devops implementation was essentially designed around empowering devs to directly provision and manage their own infrastructure, preferably via on demand automatic tools, cutting out a lot of what they believed to be dead weight middle man infra and sysops teams.  

Pretty early on they realized that if they gave devs free reign to provision their own infra they would go hog wild and blow out their budgets to the tune of tens of millions of dollars, and that devs basically didnt know a bad config from a good one because none of this was in their area of focus.  So we had various centers of excellence and specialist teams who were dedicated full time to establishing norms and patterns and implementing tooling to enforce sane policies.  I was on the team that established the enterprise monitoring policies, other people would be doing security, or network, or create reference models of how our company wrote java apps or the best practices around public API's exposed to our partners.  Where this differed most significantly from their old model was that the assumption was never that these COE teams would do the work for you.  We essentially laid out the path and the devs and app support engineers had a firehose of new things they were expected to learn about. To make this scalable the COE teams would design tools and workflows where the goal was to make the "correct" decision also the easiest and most obvious. Devs were allowed to go off the reservation and do their own thing if they really felt a need, but in 99% of cases they preferred to just deploy the approved templates and patterns and the COE teams made sure that everything in their wheel house worked as smoothly as they could manage.  

Once we had matured through the bulk of that initial policy and auditing work we basically went into the improvement part of the cycle where we just met with various teams around the company to make sure they understood how to use the existing tools and policies, and to solicit feedback on any pain points that we could improve on.  Success was measured by the percentage of our enterprise that was compliant with the existing standards, and also on how much we could reduce the amount of time the devs had to spend fussing with anything that was covered by a COE, and the unspoken metric was how much you could get your costs down.

'Click ops' was the bane of all our metrics because it means people were more likely to do things wrong, it was much less scalable and repeatable than using the templates, and by extension you were more likely to make a costly mistake because of not knowing the tricks of cloud vendor billing policies.

I really enjoyed hammering this all out while I was there.  Almost every sprint i was able to deliver some kind of improvement to the smoothness of our monitoring operations, and see the measurable successes pile up over time.  I think I thrived because we had a really open situation where I largely got to dictate what I thought I should work on, rather than getting crushed under a sea of tickets with no room to improve our processes.  It also felt a lot more collaborative working with different teams rather than throwing tickets over a wall via SNOW catalogs to a faceless team while I just waited until they were closed complete.

On the other hand I also know that at the corporate leadership level they had basically decided that it didn't matter if IT costs went up dramatically during the transition because they were expecting to be able to cut staffing and hold down later costs.  It's a lot easier to get things done when we weren't running a skeleton crew and didnt have to rationalize every drop of money or labor hours.

So I spent a couple years at a very large enterprise as they went into their devops and modernization journey.  There the devops implementation was essentially designed around empowering devs to directly provision and manage their own infrastructure, preferably via on demand automatic tools, cutting out a lot of what they believed to be dead weight middle man infra and sysops teams.  

Pretty early on they realized that if they gave devs free reign to provision their own infra they would go hog wild and blow out their budgets to the tune of tens of millions of dollars, and that devs basically didnt know a bad config from a good one because none of this was in their area of focus.  So we had various centers of excellence and specialist teams who were dedicated full time to establishing norms and patterns and implementing tooling to enforce sane policies.  I was on the team that established the enterprise monitoring policies, other people would be doing security, or network, or create reference models of how our company wrote java apps or the best practices around public API's exposed to our partners.  Where this differed most significantly from their old model was that the assumption was never that these COE teams would do the work for you.  We essentially laid out the path and the devs and app support engineers had a firehose of new things they were expected to learn about. To make this scalable the COE teams would design tools and workflows where the goal was to make the "correct" decision also the easiest and most obvious. Devs were allowed to go off the reservation and do their own thing if they really felt a need, but in 99% of cases they preferred to just deploy the approved templates and patterns and the COE teams made sure that everything in their wheel house worked as smoothly as they could manage.  

Once we had matured through the bulk of that initial policy and auditing work we basically went into the improvement part of the cycle where we just met with various teams around the company to make sure they understood how to use the existing tools and policies, and to solicit feedback on any pain points that we could improve on.  Success was measured by the percentage of our enterprise that was compliant with the existing standards, and also on how much we could reduce the amount of time the devs had to spend fussing with anything that was covered by a COE, and the unspoken metric was how much you could get your costs down.

'Click ops' was the bane of all our metrics because it means people were more likely to do things wrong, it was much less scalable and repeatable than using the templates, and by extension you were more likely to make a costly mistake because of not knowing the tricks of cloud vendor billing policies.

I really enjoyed hammering this all out while I was there.  Almost every sprint i was able to deliver some kind of improvement to the smoothness of our monitoring operations, and see the measurable successes pile up over time.  I think I thrived because we had a really open situation where I largely got to dictate what I thought I should work on, rather than getting crushed under a sea of tickets with no room to improve our processes.  It also felt a lot more collaborative working with different teams rather than throwing tickets over a wall via SNOW catalogs to a faceless team while I just waited until they were closed complete.

On the other hand I also know that at the corporate leadership level they had basically decided that it didn't matter if IT costs went up dramatically during the transition because they were expecting to be able to cut staffing and hold down later costs.  It's a lot easier to get things done when we weren't running a skeleton crew and didnt have to rationalize every drop of money or labor hours.

KMSigma.SWI said:

What does DevOps mean at your organization?  Without using the term itself, how would your company define DevOps?

I've seen two approaches:

• Use git where possible, start slowly. E.g. for some random Powershell maintenance scripts or to deploy new VMs. If those scripts are updated via git, and the boss can track them there - you're already well into the devops land. Nearly every other aspect associated with devops - like removing silos, lifecycle management, faster deployments and feedback, even culture - stem from adopting git and related techniques: central repos, forking, versioning, docs, collaboration, etc.
  (I know, simplistic - yet distills the essence of it for me. Too simplistic or even incorrect?)
• Have software devs collaborate with (or take over) (IT) ops - which mostly means using git and CM/IaC tools. This is usually faster, but I've seen this being destructive: the devs move on to their next project or lose interest (patching can be boring), and then there's nobody at the wheel.

KMSigma.SWI said:

What do you think about this shift in IT methodology?

Love it even though I have yet to fully adopt it.

KMSigma.SWI said:

What benefits can you see for your organization?

Positive tectonic cultural and resultant efficiency, productivity and reliability shifts. (Effective collaboration, way better documentation, accountability, transparency, etc.)

KMSigma.SWI said:

Where do you see barriers?

Resistance to change, existing silos, knowledge hoarding, lack of executive support, etc.

KMSigma.SWI said:

With your (fictitious) IT magic wand, what would you change?

Great question. Not sure where to start... Perhaps gently nudge everyone in IT ops to use git where possible: incentivize, train, make it fun?