cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

IT Operations: How SecOps Saves the Day

MVP

Few messages strike fear in the hearts of IT operations staff like the dreaded scan results from your security team. These messages often appear in your inbox early on Monday morning, which means it’s sitting there waiting for you. You’re not even into your second cup of coffee when you open the message:

     Good morning,

     Your servers have 14,000 vulnerabilities that must be resolved in 14 days. See the attached spreadsheet for a wall of text that’s entirely useless and requires a few hours to manipulate into actionable information.

      Love,

      Security

Sure, I’m exaggerating a bit. But only a bit.

The relationship between operations and security is fraught with tension and calamity. Often, communications from the security team indicate a lack of understanding of patching and change management processes. And equally often, communications from the operations team indicates a lack of exigency regarding addressing the vulnerabilities. But ironically, you’ll hear the same observations from both teams: “We’re just trying to do our jobs.”

For operations, the job is managing server lifecycles, handling tickets, responding to alerts, ensuring backups run, and generally juggling flaming chainsaws. For security, the job is scanning networked hosts for known vulnerabilities and missing patches, compiling reports for the executives, and disseminating scan results to the cognizant technical teams for resolution.

The problem inherent in these teams’ goals is this: there’s no clear accountability for the security of the network and its systems. Without this clarity, you’ll end up with an IT shop that is more focused on placating the other teams than it is on security.

This is where SecOps can save the day.

What Is SecOps, Anyway?

At this point, you might be thinking, “SecOps… sounds like DevOps. And that is a meaningless buzzword that belongs in the garbage heap of forgotten IT trends.” I hear you. These terms are certainly overused, and that’s a shame. Both DevOps and SecOps can solve organizational problems that are the result of a generation of siloed org charts and team assignments. And just like DevOps brings together your application and infrastructure teams, SecOps brings together security and infrastructure.

In both cases, “bringing teams together” means that the existing battle lines get erased. For example, you maybe have a security team that uses a patch scanning tool to identify systems with missing patches. That tool likely has a built-in database that can track vulnerabilities for hosts over time. It might even include a reporting function that can extract recent scan results for a subset of hosts and place that information in a PDF or spreadsheet. This are all standard, reasonable expectations from a security team’s internal tools.

On the other side of the aisle is your operations team, with their own patch management solution that has been configured to act primarily as a patch and software deployment tool. The tool can determine missing patches for managed system and act as a central repository for vendor patches. It likely has its own built-in database for keeping track of patch deployments. And it might include a reporting function. Again, these are all basic tools and capabilities of the ops staff.

What’s the problem with this approach? You’ve got valuable data sequestered from staff who could put this information to good use. And the organization as a whole can’t turn that data into knowledge. And most importantly, the focus of these two teams’ interactions will be on finding fault in each other’s data as a means to discredit the team as a whole. I’ve seen this happen. You have, too. It’s not good.

How Can SecOps Help?

A SecOps approach would open up these two datasets to both teams. Security gets a view into your ops team’s patching process, and operations gets firsthand access to scan results for the systems they maintain. And something remarkable happens: instead of the conflict that is manufactured when two teams are each working with non-shared information, you get transparency. The conflict starts to ease. The focus shifts from appeasement to collaboration. And the distrust between the teams starts to fade.

OK, that might be expecting a little much from a simple change. Because these battle lines are ingrained in the minds of IT staff, you’ll want to be patient with your new SecOps practitioners.

The next step to bring about SecOps culture change is to align the goals of each team with the organization’s goals and vision. Before SecOps, your security team’s effectiveness may have been measured by the raw number of vulnerabilities it identifies each month, or quarter, or year. And your operations team may be measured by how many tickets it closed each month, or quarter, or year. But these measurements fail to put the focus where it belongs: on the shared responsibility to create and maintain a secure IT environment. These measures are also trailing indicators, and generally suggest a fundamental problem with patching or scanning or both.

  Like all trends in IT, SecOps is no panacea for all of IT’s woes. And it requires buy-in from your technical staff who are likely suspicious of yet another “great idea” from the execs in the Ivory Tower. But with the right coaching, and the focus on SecOps as an iterative journey and not a point-in-time change, your organization will certainly benefit from bringing these teams together.

16 Comments
Level 14

Good writeup.  Fortunately it isn't an issue here.  I'm the SysAdmin and I sit next to the IT Security Manager.  He used to do my job so understands it.  I have done his job so understand it.  We fill in for each other when necessary.  We use a few tools to ensure we are reasonably up to date with patching and are reactive enough to patch serious issues immediately.  There's no conflict.  It's nice when stuff just works.  I have worked in places where there is conflict and have always tried to sort it out as it is really important to at least try to stay on top of patching.  It wasn't this difficult with Novell.       

Level 14

Thanks for the article!

Level 20

The idea of putting Ops at the end of everything really hasn't created a big change in what we are doing.  Perhaps there are some changes along the lines of agile in speeding up some cycles but the things we do haven't really changed... just the names for them have.

Level 14

True in our case as well.  I think it's really just blurring the lines a bit between teams, in regards to capabilities and permissions specifically.  At least, that's the way I view it. 

MVP
MVP

Yeah, agree. The gap between saying you're adopting a new framework and actually embracing that framework is wide. It's easy to declare, "we're doing SecOps!" and think your job is done when you set up a recurring collaboration meeting between security and operations. But you have to put in the hard work to identify the processes that are broken by sticking with silos, then rebuild those processes to take advantage of the closer teamwork between teams. I tell people all the time: don't expect perfection. That also applies to IT.

Level 13

Good Article - Thanks

MVP
MVP

There's certainly more to SecOps that sharing databases. I mentioned that as an example of the type of collaboration you need to cultivate in order to get your shop on-board with SecOps. Perhaps a bigger impact of this approach is the "baked-in vs. sprinkled-on" metaphor. When security and operations work closely together, ops can deliver new systems that are secured and compliant from Day 1. When you keep those teams separated, you'll forever live in the land of finger pointing and blame shifting.

MVP
MVP

Hey, you're welcome!

Twice a month I receive Nexpose security probe reports and scores for all of the systems for which my team is responsible.  I go through them with a fine-toothed comb and send updates about nodes incorrectly assigned to my team, correct the vulnerabilities found on my nodes immediately where possible, schedule updates to correct other nodes that will require reloads/reboots/upgrades, and identify which vulnerabilities are flat out incorrect or that cannot be corrected for any reason.

It takes time.  Fortunately NCM's Compliance capabilities allow me to stay on top of things proactively, and let me retroactively correct any discovered variances.  The ability to push out a correction on an immediate or scheduled basis to all affected nodes is a time saver.  I wouldn't want to be without NCM and its Compliance Reporting solution.

Level 15

Perfectly timed article.  We are experiencing many of the pains you describe.  Too many "the sky is falling" messages from security without an impact versus likelihood analysis and too many of these must be reviewed and completed now preventing anything else from actually getting done.  For instance, I received a notice that I needed to review and provide an action plan for a potential vulnerability.  That request ended up being 22000 items to review and I was instructed I have 2 days to complete.

I understand the culture change necessary but unless those changes start with the CIO and a standard set of expectations are not presented, the conflict will continue.  Much like the days of servers versus storage or even servers versus network.  These growing pains will need to be overcome but the vision must exist first.

Great article; thanks for sharing.

MVP
MVP

jkump  wrote:

That request ended up being 22000 items to review and I was instructed I have 2 days to complete.

Well, suddenly my 14,000 items seems like a breeze.

Level 13

Excellent write up.  We've certainly been there in the past, especially when a external team came in to do an audit and just left reams of information that took a week or two to sort out so we could even begin to start to work on it.  Thankfully that's changing.  It's a lot easier to work together and view it as a team problem and work on solutions together than to end up in a finger pointing war.

Level 16

Thanks for the write up!

If you have a decent Security Compliance Team they will do a lot of the bridge building between the teams so everyone knows their roles and expectations.

MVP
MVP

Thanks for the article

MVP
MVP

Totally agree re: decent compliance team! The challenge there is to position compliance as a partner to operations, not an auditing organization.

Level 10

Well written. We've begun embracing that here over the last 12 months or so and it has definitely helped, but we still have a ways to go.

About the Author
Long-time SolarWinds implementer and user. Spend my days now with vSphere, HDS, Nexus, and pretty much everything else.