cancel
Showing results for 
Search instead for 
Did you mean: 

Root Cause, When You're Neither the Root nor the Cause

cxi
Level 12

Expectation.png

Hey, everybody!  Welcome to this week’s quandary of Root Cause, Correlation Analysis, and having to collaborate across cross-functional teams where you have all the hands but none of the fingers!

If that sounds confusing to you, it’s because frankly, it is! I’d like to share a tale of woe and heartbreak driven by frustration in functional and equally dysfunctional IT team dynamics!

The story is set in a fairly cross-functional organization. You're probably familiar with the type. While there are clearly defined teams with responsibilities, there are also hard lines in the sand of who does what, where, when, how and why. Honestly, this story rings so true that I’ve seen this story blur with other ones. If that isn’t excitement, I don’t know what is!

As the story goes, our team had deployed a series of tools enabling a cross-stack data correlation engine allowing us to identify and truly correlate events as they happen to allow troubleshooting to be better, easier.   The problem was the true burden of responsibility this team had ALL the responsibility of identifying problems, but none of the authority to actually resolve those problems, let alone the authorization to work on them!   What makes this particularly fun is that we were chartered with and burdened by the responsibility of being held accountable for the issues until they were resolved.   If that sounds like some kind of decision made in a government sector… I wouldn’t tell you you’re wrong! J

This is where simple technical skills while essential were not good enough.  And frankly, all of the project management skills in the world wouldn’t matter here, because it’s not like a “problem” is a “project” per se.   No, we had to get everyone on board, every stakeholder at the table where egos were strong and stubborn.   Just like we discussed recently in Better Together - Working Together in Silo Organizations and When Being an Expert Isn’t Good Enough: Master of All Trades, Jack of None merely knowing the answer or the cause of the problem wasn’t good enough here.   All parties would reject the issue being theirs, even in light of evidence proving otherwise and would instead resort to finger pointing. Fortunately how we started to navigate these waters was through education of the tools we were using and how it would provide insight into their systems, access to our tools so we weren’t just the messenger they were trying to shoot but a helpful informant in things, and we also offered our guidance as IT Professionals to help them navigate the errors or problems so they could resolve them better.

It sounds so simple, it’s something fairly straight-forward but the timing it took and would continue to take whenever new members would join a team, or new problems would surface would take months or longer to reach a sense of team parity.

It’s been an interesting element of Systems Operations in the face of having intelligence, and knowledge not meaning much of anything unless you had all parties engaged, and even then that was no guarantee that people would agree, let alone do anything about it.

Have you faced a similar issue as well, where you identify a problem which isn’t your problem and the challenges faced in trying to resolve it?  Or perhaps even just having accountability for something which isn’t your responsibility and the woes of trying to get parties to take responsibility?

Or really any other story of problem correlation and root cause and how you were able to better or faster resolve it than what we faced!

16 Comments
tallyrich
Level 15

Our team works together really well when it comes to finding the root cause of issues.

Of late, however, we have been experiencing friction between a handful of the team members getting frustrated because things weren't being done "their way." The most vocal one is the kind if guy that just wants to "get it done." In fact Just Get it Done is his mantra. He sees everything from his idea of a work process and anyone that does it differently is just wrong and wasting time. Needless to say he's also the type that thinks velcro and cable management are a waste of time and who cares what it looks like if it's working. As individuals it's frustrating to work with personalities like this.

We have some on the team that are very thorough with documentation and that frustrates those that are not as thorough as they see the others as time wasters. Of course those of us that like documentation (yes, there are a few of us) we see the effort on the back side when things aren't properly documented - not to mention the issues that will be faced with audits in the future.

Then there are those that have their comfort zones and don't want to step into anything other than what they already know.

All that to say, working with people is a challenge at times.

Thanks for the thought provoking article.

cxi
Level 12

Thank you for your comments.

As someone who had to figure out "What does this un-commented code mean"..

Or who has had to try to trace a cable out only to see other cables go loose in the process (fun!)

Or... To run your hands down cable management to get your hand cut because Velcro? Whats that, we use hard plastic deep cut cable ties!

Has the friction in your environment changed due to a change in collaboration, management or leadership? Or just as time has gone on (with less and less time and more and more to do) things have started falling off the map?

news_cable_mess_03_full1.png

tallyrich
Level 15

We had a great manager that was both very technical and great with people. He moved to a different position and his position rolled to another person that wasn't as good in either area and she kept her hands off. Then we had a guy on the team take the position (the just get it done guy), but he found that it was a lot more work than he wanted to do and backed out of the position.

We've recently hired a guy that is supposed to be both team lead and hands on with the network, but he hasn't touched anything yet. And he hasn't had any one-on-ones yet. I think with the combination of all of this the team is kind of floundering in some ways.

acmtix
Level 13

I like a structured Agile approach.

Give me clear objectives, then i'll compromise with you and we will make a plan to get the job done.

Every day we will review those objectives and then decide whether our approach is working and if not we will review it and change it.

In a perfect scenario, I wouldn't have to wear more than one hat, but increasingly i'm being asked to project manage, develop, build, document and train.

The best techies are flexible, willing to learn new skills and enthusiastic. Structured approaches and technical skills can be taught later on.

The worst technical people i've encountered so far have been developers, closed to change, inflexible and difficult to negotiate with.

With a little smile and a good sense of humour anything can be achieved.

mcam
Level 14

that does sound like a nightmare

Thankfully we don't have to monitor something we don't control

byrona
Level 21

I spend part of my time in InfoSec and the other part in managing our monitoring systems so I spent a lot of time pointing out issues that I personally am not responsible for resolving.  It's incredibly frustrating pointing these out only to watch others not really care and avoid spending any time to resolve them.

vinay.by
Level 16

Good article

ecklerwr1
Level 19

I've found when working in closed area networks that since there are only a handful of people to do everything you have to know all the technology.  Some team members are better with some than others... of course.  I do monitoring, storage, networking, windows, unix/linux, security, and supervise and mentor all at the same time.  It's a real challenge but also very rewarding.  I think the limitation we have with fewer people is also a benefit in some ways because we can control the entire environment.  That exact opposite is our enterprise network where there is a large windows team, large unix/linux team, a storage team, etc.  In this instance the teams only do one specific technology for the most part.

IT_Tech_Nightmare.jpeg

mcam
Level 14

my favorite link to that picture is the management request "why can't we outsource our physical network management?"

gfsutherland
Level 14

My team uses "The buck stops here!" motto... 

- Yes it might be the app, we help the user get in touch with their support team!

- Yes their website is slow ... we will rule out issus on our end...

Tedious yes... required... .of course!

tinmann0715
Level 16

This is reminiscent of the floor under the tiles of our datacenter. No joke!

tinmann0715
Level 16

"Have you faced a similar issue as well, where you identify a problem which isn’t your problem and the challenges faced in trying to resolve it?  Or perhaps even just having accountability for something which isn’t your responsibility and the woes of trying to get parties to take responsibility?"

Darm near everyday! My career has been made on this role. I'm known as "The Wolf!" (Well, at least that is what I call myself. No one else does...)

designerfx
Level 16

This has always been true across every org. If you defend a silo/refuse to change, at some point you're going to be behind everyone else.

I've not found it being specific to any function nor age - it's more specific to a type of personality. Some of them can be eventually brought up to speed and some will defend their little kingdom to the death.

rschroeder
Level 21

Solarwinds products make discovering problems and showing responsibility so much easier!  If the parties who can correct the issues don't accept ownership from me, I just send my boss a description of the problem, the NPM/NTA graph or reports or screen shots that show the problem and its cause, and I list the team I believe is responsible for correcting the issue, and explain why they're the ones to fix the issue.  He escalates it to that team's manager, who assigns it to that team--perhaps even to the person who may have originally refused ownership.

All done courteously, respectfully, and professionally.  No ruffled feathers, just resolved problems and customers free to continue business as best they can.

nenea
Level 9

Is this a spaghetti maze?

vinay.by
Level 16

About the Author
CTO at @Xiologix - EVP of Engineering, Technology Evangelist, vExpert, EMC Elect, BDA, CISSP, MCT, Cloud, Ninja, Vegan, Single, Father, Cat, Humorist, Author