Question 7: You Get What You Measure

Recently, my colleagues , and I hosted a webcast on the topic of capacity planning and optimization. You can listen to/watch it below.

Capacity Planning and Optimization to Avoid Performance Problems

It was a really fun conversation. 

As part of the discussion, we talked about the need to measure the right things to get the correct outcome. Keep in mind my oft-repeated mantra: monitoring is simply the collection of data. You need a robust, mature tool to add context, which transforms those metrics into information. Only when you have contextually accurate information can IT folks hope to act—fixing the right problem to achieve the best (meaning most useful to the business) results.

The webcast exchange led me to wax philosophical about measuring the wrong things, which you can read here. Both of those items—the conversation and the blog post—lay the groundwork for today’s discussion. In this post, I want to talk to about metrics, measurements, and optimization (as it relates to monitoring) from a manager’s point of view.

If you accept the premise that behavior is shaped by what’s being measured; and the implied idea that managers have a significant influence on what gets measured through the values, goals, and problem areas we communicate; then the next logical question as a manager is, “How do we identify the right things to measure without a lot of trial and error?”

I think a certain amount of trial and error is unavoidable. You aren’t perfect, nor do you have the gift of prophetic vision. Perhaps more importantly, how a particular group of IT practitioners react has as much to do with team chemistry as it does the incentivizing nature of the measure-behavior feedback loop. This second point is more important because it’s the one you can do something about. But more on this idea in a moment.

There are two “laws” connected with how we think about how the things we measure effect behavior and outcomes. The first is called Goodhart’s Law:

“When a measure becomes a target, it ceases to be a good measure.”

(hat tip to Sketchplanations: https://sketchplanations.com/goodharts-law)

The second, Campbell’s Law, is perhaps a more pessimistic version of Goodhart’s

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

As an example, voting (which is important, heavily “measured,” and directly impacts social outcomes) can lead to tactical voting, gerrymandering, and even worse behaviors.

But in all the discussions of how a smart manager can run afoul of these issues one point is repeatedly skipped or glossed over: Communication. Take this example

A manager thinks there’s too much working capital tied up in excess inventory, so he measures inventory levels, or inventory turns, and “holds people accountable” for hitting their targets. The team recognizes the valuable metric—getting inventory as low as possible at the point the inventory snapshot is taken. This results in a range of creative solutions:

  • Pull forward orders from next quarter; fill and ship them early.
  • Slow down (or even stop) production in the last week or two of the quarter.
  • Shift inventory from “finished goods” to “in transit” to get it off the books.

OK, OK. The team are a bunch of slimy (possibly fraudulent) pedantic weasels, and the beleaguered manager is playing a game of cat and mouse, right? Wrong. In this example, the thing the manager failed to do was COMMUNICATE WHAT WAS NEEDED AND WHY. Furthermore, after doing so, the manager should ASK, not TELL.

Share the business goal with the team. Ask the team for ideas on how to achieve the stated goal. Even better, confirm with the team whether the goal is a sensible one or not before asking for suggestions.

If you do this, you’re more likely to get the results you want, not just the ones you measure.