A client recently asked what I thought was a pretty straight forward question: ‘How do we measure X?’ The context for the question was that a perfect metric for ‘X’ isn’t available.
I answered with a question of my own. ‘How would you describe success in ‘X’? This will determine how you measure it’. The client seemed satisfied with my response. Our discussion flowed on to other things.
But I’ve been reflecting on this question a lot over the past week.
What seemed obvious to me might not be obvious at all to others. Repeat exposure to tasks measuring impact and outcomes has taught me to measure all kinds of things. But I’m in the minority here.
What I’ve realised is that access to data is one thing. Knowing what to do with it is another.
In this article I’ll share five guiding principles around measurement. These are high-level rules of thumb that have helped me over the journey. Hopefully they will remove some of the mystic around data analysis and measurement.
1. Understand why we measure anything at all.
We measure to reduce uncertainty.
How far can we drive until the car runs out of fuel? By calculating the distance of the trip and the fuel efficiency of the car, we reduce uncertainty around running out of fuel.
Reduce is the important word here. Without knowing all variables such as wind speed and how heavy-footed the driver is, we can’t know exactly when the fuel will run out. But we’ve reduced a lot of the uncertainty.
Too often we get caught up in thinking that measurement is an exact science. It is literally the opposite.
Precise measurements are great. But if the challenge is, like my client’s question, one where there is no precise measure in place, it doesn’t automatically follow that the data you have isn’t useful.
Small reductions in uncertainty can unlock large competitive advantages and generate significant insights.
Stop looking for silver bullets and instead focus on measuring for incremental reductions of uncertainty.
2. Less is more.
I’ve written about our tendencies to over value complexity before. In the era of big data, we often think we need more data to support decision-making. That unless we have mountains of data we can’t measure.
Simplicity is still often the best case. I once had to demonstrate the impact of a policy decision to a group of bureaucrats and ministerial staffers. We had a stack of variables and created sophisticated dashboards to demonstrate the impact of the change.
In the end, none of it mattered. The audience was only interested in one measure: how many electorates would be negatively affected? We could have saved ourselves a lot of time and effort by focusing on this metric first.
A small amount of data can remove a lot of uncertainty if used correctly. What is important is that you ask the right questions (more on this shortly).
3. Horses for courses
There are measurement tasks that demand robust statistical analysis. Data science and serious research requires complex mathematical, statistical and computation skills to extract knowledge.
This should not put you off. Nor should it stop you from getting creative, rolling up your sleeves up and having a go.
The old adage of different horses for different courses applies here. If your problem requires statistical analysis use it. If not, go ahead and use the data you have to measure.
A basic understanding of data literacy will tell you when you’re out of your depth. This level of sophistication is best left to the experts.
4. Your data is only as good as the questions you ask.
Warren Berger, in his book A More Beautiful Question, argues that ‘you don’t learn unless you question’.
Berger advocates that we stop asking closed questions (How many? How much? How Fast) and learn to ask ‘open’ questions such as: Why? Why not? What if? How?
Open questions are incredibly important to understand what’s worth measuring in the first place.
My response to my client’s question was ‘how would you describe success in ‘X’?’ I framed it this way to put the onus back on the client to define the outcome they’re looking for.
Until this is defined there’s no point measuring at all.
5. Beware of paralysis by analysis.
Paralysis by analysis occurs when we have too many options and overthink a situation. William Shakespeare’s Hamlet, who appears incapable of deciding a course of action, provides a classic example of paralysis by analysis.
It’s common when there are lots of variables in play. It’s also a symptom of looking for the perfect answer. The French philosopher Voltaire cautioned in the 1700s that perfect is the enemy of the good. This holds true for measuring outcomes, which as we uncovered is not an exact science.
Despite this, paralysis by analysis happens to the best of us. For me, it’s often at a restaurant with too many mouthwatering choices!
From a business perspective, I’ve found reverse engineering the problem an effective way to guard against paralysis by analysis.
I was once hired to develop an evaluation framework for government concerning an aspirational strategy to improve adult literacy. Instead of focusing on the data, I started from the action, or decision that would drive each aspect of the strategy, and worked backwards. I asked open questions to sharpen my focus on what was required to demonstrate impact.
This process helped to clarify the goals of the strategy and determine the evidence that would demonstrate success. It also helped get past the issue of paralysis by analysis.
Measurement is a process. Not an outcome.
The goal is not to have an absolute answer, but to know more than we did beforehand. Measurement is more process than outcome. It’s a process that requires creativity and critical thinking to ask the right questions.
It doesn’t have to be perfect. It just has to be an improvement on what we currently have.
We measure to reduce uncertainty. When it is all said and done, this is what many people fail to grasp. The consequences are significant. We end up striving for certainty and either measure the wrong things or, worse still, don’t measure at all.
Start by asking good questions to define your challenge. Determine the data at your disposal to provide an answer, no matter how small or insignificant the data might seem.
And don’t overthink it!
