Change is Hard

I.T. folks understand change can be hard, and I don’t mean in the “Who Moved My Cheese” kind of way. Change, as in “change control,” is necessary. But we don’t like the process any more because we understand how essential it is. For those who haven’t experienced the Kafkaesque confusion that is corporate change control, here’s a snippet of a chat between coworkers to illustrate the point:

Adato, Leon‎‎ [1:52 PM]: Is Joe covering for us in the change meetings from now on?

Sharp, Matt [1:59 PM]: Not exactly. He’s there because he is a change coordinator for a lot of groups. BUT he doesn’t know enough about the changes to describe them in technical detail. So each of us need to be on the call to talk about the technical details if we have a change.

It’s a colossal waste of time because there is no order to how they proceed through the changes. Sometimes we go in the first five mins of the call.

Best part: If you miss the window then you have to wait 90 mins for them to loop back around for another chance to discuss.

Even better: They give you about 5 seconds to pipe up about your change or they move on and deny it.

‎‎Adato, Leon‎‎ [2:02 PM]: Wow. Nothing like bureaucracy at its finest.

Sharp, Matt [2:04 PM]: yuuuuup. I was five mins late two weeks ago then had to wait until 10:57 to discuss; then had to be on the 11am call to explain all over again; then the 11am call told me I had to call the Mgmt Business Team Sit Rep line. Then they told me I had to send a detailed email to an address documenting the change.

...then they denied it due to short notice

...then Ben stepped in and upgraded the change

...then the change people approved it but the MBT change overlords denied it

...then the VP of the division stepped in and said, “do it” and everyone STFU

...and i was able to make the change that night.

Of course, all of this came to a quick conclusion at 4:30pm just after I had sent an email to the vendor letting them know that the change was canceled

then at 4:35pm it was back on.

So I had to apologize and return to regularly scheduled. Then I had to work that night from 9pm to 1am to actually perform the change I spent 8 hours wasting to get approved

The punchline? This was all to fix a negative value for CPU for one node because it was on the MBT dashboard and bothering an exec.

<end scene>

Navigating the special bureaucracy of the CRB (change review board) is one of the more frustrating experiences we face when building a career in tech. My point in discussing it here is to prove that I—and everyone here at SolarWinds—understand what we’re asking of customers when we publish an important upgrade—such one we may have posted recently, for instance.

SolarWinds has always prided itself in—and worked to be true to—it’s Geekbuilt^® heritage. We create solutions to solve problems we’ve experienced, that work in ways we appreciate when we’re in the field (or the NOC, as it were). Part and parcel of this is heightened sensitivity of the hurdle upgrades, patches, and hotfixes represent inside the machinery of your business. Sure, we’re always super excited for you to get your hands on the latest features—because we worked hard to build them, and because they address issues you told us about in the first place. At the same time, rolling those upgrades into product isn’t trivial for most of our customers, so we temper our enthusiasm with restraint.

However, (as the news cycle loves to constantly remind us) we live in unprecedented times, and we’re adapting to meet the realities of the challenges we now face.

In the short term, that means emailing our entire customer base to encourage, assist, and yes, even cajole you into upgrading to the latest version of our tools—both to protect yourself from a certain incident that came to light last December; and to ensure you don’t receive error messages when the old digital certificate used to sign our files expires, in lieu of the one in the updated versions.

Looking ahead, it means embracing our “Secure by Design” philosophy by providing regular updates you can plan for; and a more security-centric explanation of what issues (such as CVEs) each upgrade, patch, and hotfix addresses. If nothing else, we’re doing it so you have as complete a picture as possible, in the hope it’ll help speed you through the unavoidable reality (and yes, I’ll say it, unbearable frustration) of change control meetings.