cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

There’s Always That One Guy… How to Build Team Accountability and Reduce Firefighting.

Product Manager

Change control. In theory it works. However, there’s always one person who thinks the process doesn’t apply to them. Their justification for going rogue may sound something like, “There’s no time to wait, this has to be done now,” and, “This is a small change, it won’t impact anything else,” or maybe, “This change will make things better.”

But at the end of the day, those changes inevitably end up crashing a service, slowing application performance, or even worse, opening new vulnerabilities. The call will come in, something’s broken and magically no one will know why on earth it’s happening and, they certainly won’t be able to remember if any changes occurred…or who made a change. There goes the rest of your day, looking for the root cause of an issue created by one of your own coworkers.

Recently, Head Geeks Thomas LaRock sqlrockstar and Leon Adato adatole hosted a THWACKcamp session on this exact topic. In their scenario the culprit was “Brad the DBA.” At SolarWinds, we understand this all-too-common scenario and have a tool designed to help.

SolarWinds® Server Configuration Monitor (SCM) provides an easy-to-use and affordable way to track when server or application configuration changes are being made, who’s making the changes, and what the differences are between the old configuration and the new configuration. It detects, tracks, and alerts on changes to things like hardware, software, operating systems, text and binary files, Window Registry, and script outputs on Windows® and Linux® servers.

pastedImage_0.png

Additionally, SCM is an Orion Platform-based module, meaning you can quickly correlate configuration changes with infrastructure and application performance metrics in a single view. Helping confirm or illuminate the possibility of a configuration change being the culprit.

pastedImage_1.png

These capabilities help provide you with the visibility needed to not only remediate issues faster but, also hold non-process-abiding team members accountable for their actions. If you’re tired of the shenanigans created by your colleagues not following the change control process for your servers and applications, check out a free, 30-day trial of Server Configuration Monitor. And just for fun, if you have a good story of how “Brad” broke your day, feel free to share below!

7 Comments
Level 13

So true.  Hopefully you only have one of those.  They always have some super funky way of "making it work" that doesn't work with anything else and causes all kinds of problems in other areas (which don't effect them so they are fine with it - they just put an SEP field around it).

Using NCM was a great way to give staff incentive for using Change Management / Change Control.  NCM simplified the task of learning what changed and when.  Combining it with NPM's log for Solarwinds administrators who were active at the time, and Cisco TACACS Reports to show who logged in, at what time, and issued which commands, really shows people their actions are no longer anonymous or hard to discover.

It's hard to give up a habit of "cowboy networking" where changes are made at any time by any network analyst who feels the work needs to be performed immediately and without documentation/permission/notification.

Level 15

I agree with rschroeder​ on the benefits of NCM.  The real-time reporting has made things easier to understand when changes happen and who made the changes.  The side-by-side view also allows understanding of what was changed.  I am looking forward to seeing if SCM will provide the same benefits. 

Product Manager
Product Manager

Hi jkump​!

Thanks for your post. I am very happy to hear NCM is a valuable tool for you. The items you mention (real time reporting, who made the change, and side-by-side comparisons) are all features that exist in SCM as well. In fact, the latest release of NCM borrowed code from SCM to better show the side-by-side compare. Real time alerting/reporting on changes and who made the change are available for Windows servers and application via the Orion Agent.  Below are a few screen shots from SCM:

who made the change 2.PNG

Compare and who mande the change.PNG

Happy to help if you have any questions.

Level 16

We have a pretty strict change policy. First time you make a change without a change control you get a talking to from your boss. The second time you get a box.

MVP
MVP

I'm a big proponent of change control - but it needs to be well thought out, organized and efficient. I worked with a company once that the initial go-round of change control could take up to 2 weeks to get a change approved - that didn't work for my organization as our customer was used to requesting a change in the morning and having it implemented the same day. They later revised that policy and it became much more efficient and my customer then bought into it. Yes, with CC it now took a couple of days to implement a change, but that was an acceptable price as opposed to weeks.

Level 11

working in an MSP environment leads to all kinds of cowboys "fixing things" on the fly and logging almost nothing.  DAYS spent tracking down the change, why it happened and is it actually/still needed.  sometimes the change was "I was trying it out to see" or "I thought I cancelled that change" or some other nonchalant comment followed by "if you want to change it back go ahead".  burns my toast that does.