cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Troubleshooting: A Forcing Function on the IT Career Path

Level 13

Troubleshooting efficiency and effectiveness are core to uncovering the root cause of incidents and bad events in any data center environment. In my previous post about the troubleshooting radius and the IT seagull, troubleshooting efficacy is the key performance indicator in fixing it fast. But troubleshooting is an avenue that IT pros dare not to walk too often for fear of being blamed for being incompetent or incorrect.

We still need to be right a lot more than we are wrong. Our profession does not give quarters when things go wrong. The blame game anyone? When I joined IT operations many a years ago, one of my first mentors gave me some sage advice from his own IT journey. It’s similar to the three envelope CEO story that many IT pros have heard before.

  1. When you run into your first major (if you can’t solve it, you’ll be fired) problem, open the first envelope. The first envelope’s message is easy – blame your predecessor.
  2. When you run into the second major problem, open the second envelope. Its message is simply – reorganize i.e. change something whether it’s your role or your team.
  3. When you run into the third major problem, open the third envelope. Its message is to prepare three envelopes for your successor because you’re changing company willingly or unwillingly.  

A lifetime of troubleshooting comes with its ups and downs. Looking back, it has provided many an opportunity to change my career trajectory. For instance, troubleshooting the lack of performance boost from a technology invented by the number one global software vendor almost cost me my job; but it also re-defined me as a professional. I learned to stand up for myself professionally. As Agent Carter states, "Compromise where you can. And where you can’t, don’t. Even if everyone is telling you that something wrong is something right, even if the whole world is telling you to move. It is your duty to plant yourself like a tree, look them in the eye and say, no. You move." And I was right.

It’s interesting to look back, examine the events and associated time-series data to see how close to the root cause signal I got before being mired in the noise or vice-versa. The root cause of troubleshooting this IT career is one that I’m addicted to, whether it’s the change and the opportunity or all the gains through all the pains.

Share your career stories and how troubleshooting mishap or gold brought you shame or fame below in the comment section.

19 Comments
MVP
MVP

Love the envelope theory - sometimes it is quite true.

Level 13

I love reading Kong's posts...

Ok, I'm going to get blamed for promoting my troubleshooting story again, but so what...here goes

🙂

MVP
MVP

Nice

Level 16

After being a Network Architect for many years and hearing 'it's the network' one too many times I jumped over to Enterprise Monitoring team for a change. (Envelope #2)

In that role I found myself still being involved in every major issue, but not in a defensive position. As part of the Monitoring Team we always tried to assist in troubleshooting using all of the tools we owned, then once resolved implement additional monitoring to provide early warning if the issue returned.

I opened my Envelope #3 when my former employer asked me to replace my Solarwinds environment with a different product but still maintain all of the same functionality as NPM, SAM, NCM, IPSLA, and Netflow. 

Level 20

Having a thorough understanding of multiple technologies and then being able to apply to troubleshooting real difficult problems is one of my best skills and it's also one of the things that makes me enjoy my line of work.  Sure it's really tough and rough sometimes.  You can't always win... but the wins are really self fulfilling.  Not being afraid to dig in and investigate, collect data, correlate time series, and ultimately find the root cause is very rewarding.  This is what made me stay in the field of communications, security, and infrastructure design, maintenance, and troubleshooting.

Level 14

Love the envelope story... and it hit home in many,many ways. I've experienced all three!

I often joke that I've made a career "cleaning up after the elephant...." (see envelope 1)

I am currently experiencing envelope 2...

all because... 2.5 years ago... envelope 3 happened.

kong.yang​ outstanding post.... thank you!!!

Nicely done!

The envelopes remind me of a tiny little (fictional) book called "How To Succeed In Business Without Really Trying."  It's actually a broadway musical that was turned into a hit movie back in the early 1960's, and that little book is full of wry advice that the protagonist in the play/movie follows literally--to the consternation of his coworkers and the hilarity of the audience.

There was a recent remake of the movie, featuring Matthew Broderick, but the original 1963 movie starred Robert Morris, who was the inventor of the character on Broadway, and that movie's the best!  Check it out if you have a chance--it's a spot-on tongue-in-cheek analysis of business and male chauvinistic behavior couple with cronyism and nepotism.  All done in a very entertaining manner, coupled with great music, period costumes, and fun music.  And the obligatory love story.

You'll see a lot of what still goes on today in business, and you'll recognize folks around you from the movie.

Level 10

I have seen many people use the envelope story. It is not so funny if you are at the receiving end.

MVP
MVP

I'd never heard the envelope story, I like it! But none of it has applied to me. I was in my first job for about 6 years and I'm about to notch up 22 years in my current job. So I've not ever opened up any envelopes. I left my first job to go to my current one for increased wages. I've been blamed for things over the years and either they've been wrong or right. I've gone with it whatever the case was. Unfortunately mistakes are made at times. Only way to learn right?

Level 13

yeah, me neither, but you're right - it is good.

Level 13

Open the 4th envelope -- "Its time to retire"

MVP
MVP

Never heard the envelope story before...Hmmm.

I will need to check out rschroeder​s movie recommendation.

Level 10

It’s interesting to look back, examine the events and associated time-series data to see how close to the root cause signal I got before being mired in the noise or vice-versa. The root cause of troubleshooting this IT career is one that I’m addicted to, whether it’s the change and the opportunity or all the gains through all the pains.

Great post!  Love this statement!  I'm addicted to IT as well

Level 14

I have always loved troubleshooting.  It has made me what I am today.   It has projected my career forward so many time.  I do most of my learning based off of troubleshooting and the need to know WHY. 

Level 9

Even when nothing can go wrong it does. To expect more then that is unrealistic. Troubleshooting is part science, part intuition. Often it's impossible to document or even explain the intuition part. Everyone wants a root cause, and the reassurance that it will never happen again. Yet it's true even when nothing can go wrong it does.

New story to me as well.

One thing I say on a group phone call or team meeting when troubleshooting is "I am willing to be wrong (but I can't find the issue) ".  I do this for the following reasons.

  •      It could be on my companies side, like the infrastructure, config or whatever....and I want to know if they can assist me.
  •      I hope it makes the others think that "If he doesn't think the problem is on his side, I better double check to make sure I am clean with my part."
  •      It is also my way of saying, "Please help"

Many times I found out it is a database that we control is sick.  I thank the App Dev group for their help and work with the DBAs to get it better.

Now that we have DPA....we have one less thing to guess about.

RT

Troubleshooting someone else's problem is particularly enjoyable for me.

But I don't mind troubleshooting my own company's issues, as long as I can get it right relatively quickly.  And being recognized for being helpful is always a bonus, for me.

Level 12

this is sadly true. i got demoted as and helpdesk scheduler from a it system administrator. i'm looking for another job. ;(

MVP
MVP

Good troubleshooting skills are what separate the excellent from the average. Anyone (relatively speaking) can run a well functioning network - it's when things go awry that skills are really needed.

It reminds me of the old story (and there are a ton of variations) where the consultant is called in to fix some expensive piece of equipment. The consultant examines the device and thinks for a minute then takes a hammer and hits the device which immediately begins working again. When the boss is handed a bill for $10,000 he balks at the price stating that all he did was to hit it with a hammer and anyone could have done that. He returns the bill and demands justification. The contractor revises the bill - Line 1) hit device with hammer $1 Line 2) knowing where to hit the device $9,999.

That summarizes good troubleshooting - maybe anyone could "hit it with a hammer" but how much money is lost during the time it takes for anyone to figure it out.