Change is Hard

I.T. folks understand change can be hard, and I don’t mean in the “Who Moved My Cheese” kind of way. Change, as in “change control,” is necessary. But we don’t like the process any more because we understand how essential it is. For those who haven’t experienced the Kafkaesque confusion that is corporate change control, here’s a snippet of a chat between coworkers to illustrate the point:

Adato, Leon‎‎ [1:52 PM]: Is Joe covering for us in the change meetings from now on?

Sharp, Matt [1:59 PM]: Not exactly. He’s there because he is a change coordinator for a lot of groups. BUT he doesn’t know enough about the changes to describe them in technical detail. So each of us need to be on the call to talk about the technical details if we have a change.

It’s a colossal waste of time because there is no order to how they proceed through the changes. Sometimes we go in the first five mins of the call.

Best part: If you miss the window then you have to wait 90 mins for them to loop back around for another chance to discuss.

Even better: They give you about 5 seconds to pipe up about your change or they move on and deny it.

‎‎Adato, Leon‎‎ [2:02 PM]: Wow. Nothing like bureaucracy at its finest.

Sharp, Matt [2:04 PM]: yuuuuup. I was five mins late two weeks ago then had to wait until 10:57 to discuss; then had to be on the 11am call to explain all over again; then the 11am call told me I had to call the Mgmt Business Team Sit Rep line. Then they told me I had to send a detailed email to an address documenting the change.

...then they denied it due to short notice

...then Ben stepped in and upgraded the change

...then the change people approved it but the MBT change overlords denied it

...then the VP of the division stepped in and said, “do it” and everyone STFU

...and i was able to make the change that night.

Of course, all of this came to a quick conclusion at 4:30pm just after I had sent an email to the vendor letting them know that the change was canceled

then at 4:35pm it was back on.

So I had to apologize and return to regularly scheduled. Then I had to work that night from 9pm to 1am to actually perform the change I spent 8 hours wasting to get approved

The punchline? This was all to fix a negative value for CPU for one node because it was on the MBT dashboard and bothering an exec.

<end scene>

Navigating the special bureaucracy of the CRB (change review board) is one of the more frustrating experiences we face when building a career in tech. My point in discussing it here is to prove that I—and everyone here at SolarWinds—understand what we’re asking of customers when we publish an important upgrade—such one we may have posted recently, for instance.

SolarWinds has always prided itself in—and worked to be true to—it’s Geekbuilt® heritage. We create solutions to solve problems we’ve experienced, that work in ways we appreciate when we’re in the field (or the NOC, as it were). Part and parcel of this is heightened sensitivity of the hurdle upgrades, patches, and hotfixes represent inside the machinery of your business. Sure, we’re always super excited for you to get your hands on the latest features—because we worked hard to build them, and because they address issues you told us about in the first place. At the same time, rolling those upgrades into product isn’t trivial for most of our customers, so we temper our enthusiasm with restraint.

However, (as the news cycle loves to constantly remind us) we live in unprecedented times, and we’re adapting to meet the realities of the challenges we now face.

In the short term, that means emailing our entire customer base to encourage, assist, and yes, even cajole you into upgrading to the latest version of our tools—both to protect yourself from a certain incident that came to light last December; and to ensure you don’t receive error messages when the old digital certificate used to sign our files expires, in lieu of the one in the updated versions.

Looking ahead, it means embracing our “Secure by Design” philosophy by providing regular updates you can plan for; and a more security-centric explanation of what issues (such as CVEs) each upgrade, patch, and hotfix addresses. If nothing else, we’re doing it so you have as complete a picture as possible, in the hope it’ll help speed you through the unavoidable reality (and yes, I’ll say it, unbearable frustration) of change control meetings.

Anonymous
  • And that's how it ought to be! I'm willing to bet this comment may get you more than one request of "can you show me how to do it at my company"?

    And I hope that helps everyone avoid the more horrific aspects of Changes.

    But in the meanwhile, GET YOUR ORION UPGRADE SCHEDULED!!

    ;-)

  • Maybe because I am a bit Geeky. But when I played a major part in our Change Control process at my last job it started out REALLY awful. Just like you said. It was tough to get people engaged, they didn't want to show up, meetings were hours long. But we kept pushing. We got to the point that if you performed a change that was not scheduled and not related to an incident, you had your privileged credentials disabled. If that meant you couldn't get a job done, well that would be reflected in your performance appraisal. We only had to do that once, to one person, and everyone knew we meant business. Within 18 months or so our Change Control review meetings took 15 to 20 minutes. Included some 50+ people on the phone and in person. We had all the stakeholders who could say NO NOT NOW to a scheduled activity. What a CCRB meeting was not was a design meeting. It was not a project meeting. If most people in the meeting didn't already know what you were doing before we met then some communication channel broke. All the architecting, documentation, technical debate should have occurred before. Sure all the technical people were there to answer questions but it was mostly about the interactions or impact it would have on specific stakeholder services during the change event.

  • Your story sounds all too familiar.  Sometimes a business can feel too large for those responsible for getting timely and effective changes through Change Management. The size can slow down information sharing & understanding, which slows down Change Approval.  Worse, there may be too many gun-shy Managers, or too few technical experts with excellent communication skills to get the idea across in ways that instill confidence in those voting the change up or down.

    Long ago, I went from a business with a few dozen employees, to one with three thousand, and finally to one with 17,000.  Each step up brought more responsibility (and access to cool and powerful applications & hardware), but it was always accompanied by increased inability to perform changes in a short enough period to effectively mitigate potential problems.

    I likened it to driving a small rowboat with a little motor, compared to driving a tour boat with a few dozen passengers, to captaining a 1000-foot freighter on the Great Lakes. 

    The rowboat can stop & go & turn very quickly, just like a many-hats I.T. person can make changes in real time.  

    The tour boat operator has to coordinate with deck hands for tying up or untying the boat at the dock, as well as give extra time for stopping due to the increased inertia, and is responsible for the safety of all those passengers.  There's no quick stopping & starting, lest people fall down (or overboard!).

    The freighter captain knows it's physically impossible to quickly stop or turn a vessel containing 70,000 tons of ore or coal.  It make take between one and three miles to stop that load, depending on the speed it was at when an emergent situation mandated stopping it.  And it could take a mile or more to turn it around, depending on wind & waves & weather.

    As one moves to larger organizations, one must be aware that rapid changes that aren't vetted by experts and shared with all affected customers are no longer possible.  I call those rapid changes "Cowboy Networking", after James T. Kirk's "Cowboy Diplomacy".  It's easy to get used to doing quick changes when YOU want, and hard to learn to postpone them for weeks or months until a satisfactory number of departments and people have come to understand the impact, and given you their maintenance windows (which you have to coordinate with the many different departments), and finally vote to let you make your change.

    Even when the change isn't expected to bring the network down.  

    That's the price of doing business.  Especially in critical health care networking.

    So yes, "change is hard."  Learn the ways to facilitate making change easier for others.  Get your mindset focused on planning instead of immediately doing.  

    And you won't be accused of practicing Cowboy Networking.

    Swift Packets, all!

    Rick Schroeder

    In the Little Red House

    In the Saginaw Wood

    p.s.:

    This is the Paul R. Tregurtha, a Great Lakes freighter that typically hauls coal or iron ore (taconite) to or from my home port of Duluth, Minnesota, on the western tip of Lake Superior.  https://en.wikipedia.org/wiki/MV_Paul_R._Tregurtha

    1013 feet, 6 inches long.  (The owners required the ship be built six inches longer than the other twelve thousand-footers on the Great Lakes, so she'd hold the title of Queen of the Lakes--reserved for the biggest boat out there)

  • Leon... I could not help but groan, nod my head and snicker as I read this. The larger the process, the longer the time to change. The balancing act is to make the business better and more efficient versus the needed checks, balances, code reviews, back out plans and appropriate notifications. Change Control is the living definition of the term: Necessary Evil!