Are we not using data to drive decisions? [Unscientific Survey Results]

KMSigma.SWI said:

Is your organization using data driven decision making?

Financial data - probably in good shape. IT and UX data - probably 10% of what's possible and before we could say "100% data driven" as opposed to anything else.

KMSigma.SWI said:

If it's not 100% data driven, what percentage is based on "gut" feelings?

"Gut" as in, based on precedent, education, brainstorming with peers - in addition to inexplicable intuitive urges?  Probably lots.

KMSigma.SWI said:

If there are gut feelings, where do you think this comes from?

Kinda like LLMs with their "next word prediction" mechanism: "I can't explain how I arrived at this but based on everything I learned in the past (aka my training set), here is what I think the next step (word) should be". Except in humans, it's way less predictable: mood, caffeination level, how things are at home and at work, office politics, personal agendas, all that.

You're embarking on one of the key questions of 21st century IT, KMSigma.SWI . Most of the Big4 consultancies along with the big management consultancies (e.g. McKinsey, Bain, BCG, et al) spend a lot of time and energy researching exactly your hypothesis. 

Here's an example of a very recent and informative report from McKinsey on organizations who are succeeding at the cultural shift to data-driven decision making https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-data-driven-enterprise-of-2025#/

I'd love to discuss this topic further! Cheers,

-Kevin

This is why I'm asking.  Just because "it's always been this way" has never set will with me. I'm the five year old in the back always asking "why?"

This can make me very annoying in meetings, but when I'm provided a quality answer, everyone in the room benefits.

kekline said:

You're embarking on one of the key questions

Kevin: can you help? (honest question) What is the question?

• Whether a business uses a structured approach to data-driven decisions?
• How to get there?
• How to determine whether, and to what degree, a business is data-driven in its decision making across its entire operation?
• To what degree decision making is data-driven in a specific area? (e.g. a product, or specifically, incident response, capacity or development planning for that product)

Take Google's core application like maps and if we ask ourselves whether they use data to make decisions about it - such as inform development efforts or allocate compute capacity, or decide whether to hire another SRE - the answer is likely, "sure". They've got their failure rates, error budgets, focus groups, user surveys, anomaly detection and all that - yay! (Is their decision making 100% data driven? 99%? Hard to say, yet it's still likely it's mostly data-driven.)

Then ask whether the hiring of that SRE is data-driven, and the answer is highly unlikely to be a 100% "yes", because at least some of the decision making will be based on "who do we know who is good?" and not on data. Data-driven approach will be something like:

• out of the pool of known / available SREs or SRE-aligned techies, get 10 scoring the highest on the availability and "biggest bang per buck", and lowest on the failure rate metrics.
• ensure the metrics represent the SLIs really well, i.e. that the available data and the decision process are highly accurate, efficient and effective in selecting the right candidate (this is super hard - humans aren't distributed web apps and it's not easy to collect the "four golden signals" about them) 

The actual criteria will more likely be along these lines:

• Does the HM know anyone? Anyone knows anyone?
• Engage the recruitment team. Give them the same alphabet (ha!) soup JD we gave them last time.
• Selecting candidates from the internal pool: let the person with the most glowing performance review win! (Is that person truly a data-driven best match with 99% confidence? Ha!) External candidates: let the best resume win! (Good match? Ha! :))
• Interview the hell out of them. Last man or woman standing - subject to further torture. If all fails - fire the recruitment team and I'll just go post the job on LinkedIn.

... and if we assume Google's hiring process is maybe 50% data-driven (too many human factors in play, the interview process and the selection criteria less-than-perfectly aligned with the actual job and how a given candidate will perform) - then what does it say about other companies?

... and if:

• human capital is one of the biggest factors contributing to company's viability or success, and
• the candidate selection and hiring process is not very (or fully) data-driven and unlikely to be anytime soon

... then what does this say about data-driven decision making at even the most technology- and AI-driven companies?

(Sure, I picked one of the most challenging areas for data-driven anything - the point is, if we try to gauge or score company's entire ops on data-driven decisions, we have to include those challenging areas as well - and I haven't even touched on the disconnect between the share value and that of the company.)

Anecdotally I have seen companies where data was not really leveraged well for a variety of reasons.

Gaps in the originating the data - you built a hot new product but the devs were focused on just getting the features ready to ship, they didn't think ahead and implement the right kind of telemetry to capture critical parts of the user experience and behaviors.  Now we have to choose between having devs go back and rework old code to add necessary instrumentation versus shipping new features.  Better internal instrumentation rarely gets the sales org and potential customers hyped.  It's even more challenging with non-software businesses where getting the data you want usually means interrupting a human's workflow so they can record some data point that they feel is entirely meaningless to their personal responsibilities.  When I worked in restaurants cooks are supposed to be routinely tracking the temps of cold items and recording it for health inspectors, but you can rest assured that a lot of places those checks do not happen at all and the cook just pencil whips random numbers onto a piece of paper at the start of their shift before they get busy doing their "real job" of actual cooking.  Most employees struggle to tell you how much time they spent on individual tasks, even in consulting where they have to bill customers based on the time spent working for them.  Often they dont record time until Friday, and by then you get very loose approximations of how long their gut told them they were working on that thing, if not completely made up numbers to ensure that they just hit the even 40 their manager wants to see.

Pain when trying to access the data - Is the data still on hand-filled paperwork that then needs to be processed into some kind of computerized system to ever be useful?  Is the data processing error free and fast enough to drive real time decisions? Can't tell you how many times I walked into data that was just an ocean of free text boxes where people put any number of special characters or inconsistent inputs that break when I try to turn it into structured data. Is the database available to all the people who might have a need for it to drive their decisions? Do those people even know how to construct queries or do they have to pitch requests to a business analyst who has the query language skills but likely doesn't understand the business context? So as they explore the available tables and columns they don't know that it would be helpful or not if they included certain attributes.  Maybe the fields that the business person would just naturally assume should be captured aren't in the raw data so now we have to circle back to the previous problem and improve our data collection.  But the decisions need to be made now, not in 2 quarters when the changes get implemented and we have time to collect samples.  Also is it performant and cost effective to even be able to query the data set? With the shift to -aaS paying by seat count is very common and companies often decide to ration out who has access to begin with, and then if you don't know what you are doing you can write a query that literally costs hundreds of dollars to execute and then realize you forgot to include a field you would need so you get to modify the query and incur that cost again.  With big data sets and labor costs factored in sometimes a good report legitimately cost tens of thousands of dollars to produce.  I worked with a company where raw data would be stored natively in our software, then a massaged version of that was shipped to Snowflake, then yet another layer of massaging as it was turned into leadership dashboards in Looker.  At each transformation point people were making decisions about what data they thought the layer above would be looking for, but since I was one of the few people with access and expertise to cross reference raw data to the exec level versions I would often be in meetings and have to call out that the Looker version was misleading/incomplete in some critical ways that was leading the audience to conclusions that didn't align with the on-the-ground reality I was seeing when I worked directly with our users.  Maybe we have time to let me reassemble the queries to show another story, maybe we need an answer by 2pm.  So in the end business leaders, especially those who are more executive and strategy instead of technical and detail oriented, end up having to just make what they think is the best decision hoping the quality of what they have been given was up to par. 

Personalities - At that level there also risks being some serious egos driving what they want to see happen.  If the data conflicts with their priors, and they have any reason to question the data then it's easy to just disregard it, or apply some sort of mitigating excuse for why the numbers look like this, but they know that we need to do that.  I once worked with a very technical leader who was just the grand master BS artist.  He knew how to manipulate the people above and below him to a level where the company was willing to redirect hundreds of millions of dollars in payroll toward him and the teams he led and away from any other potential competing work managed outside his sphere of influence.  This happening despite some other people in the background raising very legit doubts that any of his projects were actually profitable to the organization.  He was the kind of guy I knew I had to record everything he said in meetings because he would promise the world in person and make it seem like his vision was the perfect solution and our exec team fell for it over and over for several years until we finally hit a critical mass of failures to deliver results and they cut him loose.  On the power of his charisma he led the execs to make several decisions in spite of contradictory or incomplete data and essentially derailed years of development because he was so certain that his vision was the best.

It's fascinating to see the emphasis on Enterprise Technology Management and Data Analytics in your survey. Data-driven decision-making tools are crucial for bridging the gap in how companies utilize their data. I'd love to hear more about the specific tools and strategies your fellow IT professionals are using to enhance their data-driven approaches!