Before You Trust the Measure

Business leaders depend on measures. Sales are up 11 percent. Customer satisfaction is 87 percent. Productivity improved 6 percent. Market share increased two points. Put those numbers on a dashboard, add a few charts and arrows, and they acquire an impressive sense of authority. But before acting on a measure, there is a question worth asking: What exactly are we measuring? That sounds obvious. It often isn't. Suppose a company reports that sales increased 12 percent during the past year. That seems like good news. Now suppose we look underneath the number. Unit sales declined 4 percent. Average selling prices increased 17 percent. Direct costs increased 15 percent. Sales really did increase 12 percent. The number is accurate. But the business story is considerably more complicated than the number suggests. The company is collecting more dollars while selling fewer units. Its price increases are being partly consumed by higher direct costs. Depending on the business, declining unit volume may be far more important to management than the increase in nominal revenue. Nothing was wrong with the sales figure. The problem was what we thought it told us.

A Number Has a History

Every business measure comes from somewhere. Something happened in the real world. Someone or something observed it. The event was recorded. The record entered a system. It was classified, combined with other records, transformed by calculations, and eventually presented as a measure. At every step, choices were made. Consider something as ordinary as customer retention. What is a customer? Someone who purchased during the past month? The past year? Anyone with an active account? What counts as retained? One additional purchase? An active subscription? No cancellation notice? What happens to customers who bought a product they are unlikely to buy again for five years? The calculation can be mathematically flawless and still produce a measure that means something quite different from what a manager assumes when reading Customer Retention: 91%.

The same questions apply to productivity, quality, customer satisfaction, employee turnover, market share and hundreds of other measures businesses use every day. Before trusting a measure, we should understand its provenance. Where did the information come from? How was it collected? What was included and excluded? How was the measure defined? How current is the underlying information? Has the definition changed? And, perhaps most importantly, does the measure actually represent the thing we think it represents?

The Picture Is Part of the Information

There is another layer that receives less attention. How information is displayed can change how we understand it. Consider the humble pie chart. Pie charts are very good at communicating one particular idea: this is a portion of a whole. A division represents 25 percent of company sales. A product represents 15 percent of the market. A department receives 10 percent of the budget. The geometry is immediately understandable. But the geometry also communicates something else. The circle is a fixed whole. If one slice becomes proportionally larger, another must become proportionally smaller. That is exactly right when we are displaying how a fixed $10 million budget was allocated. There really is a fixed whole. It may be a poor mental model for a growing business. Suppose four divisions maintain exactly the same relative shares of company revenue while every division grows 50 percent. A pie chart for the beginning of the year and another for the end could look identical. The company grew 50 percent, yet one of the most important facts about the business has disappeared from the visual representation. Scale creates a similar problem. Twenty percent of a $10 million market and 20 percent of a $1 billion market can be represented by identical circles and identical slices. The percentages are correct. The picture is correct. Yet something important has been lost. Every visual representation emphasizes some information and suppresses other information. That doesn't make visualization bad. It means the choice of visualization is itself part of communicating trustworthy information. We should ask not only: Are the numbers in this chart correct? We should also ask: What will someone naturally conclude from seeing this chart, and is that conclusion justified?

Measures Are Evidence, Not Objectives

This distinction becomes particularly important with performance measures. Imagine a customer service organization trying to improve productivity. Management chooses average call handling time as a key measure.

The current average is nine minutes. The target is eight. Employees are trained. Processes are changed. Supervisors concentrate on call duration. Six months later, average handling time is seven minutes and forty seconds. Success. Perhaps. What happened to first-call resolution? Did repeat calls increase? Did customer satisfaction change? Did employees rush customers off the telephone? Did total customer-service costs decline? Average handling time may be a useful measure. But it isn't necessarily the objective. It is evidence about the objective. Once an organization forgets that distinction, it can become very good at improving a number while making the underlying business worse. Economists and management researchers have warned about this problem for decades. Goodhart's Law is often summarized as: when a measure becomes a target, it ceases to be a good measure. But AI introduces another concern. The measure doesn't even have to become a target. It can simply become an input.

Then Comes AI

Artificial intelligence makes this problem considerably more important. Businesses are beginning to use AI to analyze performance, identify patterns, detect anomalies, forecast results and recommend actions. Increasingly, AI will not merely present information to managers. It will help interpret that information. Eventually, in many applications, it will act upon it. That creates an enormous opportunity. It also means we need to become more careful about the measures we give these systems. Suppose an AI is asked: Analyze our customer-service operations and recommend ways to improve productivity. The company's accepted productivity measure is calls handled per employee-hour. The underlying data can be pristine. The AI can analyze it perfectly. It may discover patterns humans missed. It may identify process changes that genuinely increase calls handled per employee-hour. And it can still lead the company in the wrong direction. Perhaps what the business actually needs is more problems resolved on the first contact, higher customer retention and lower total service cost. AI does not need bad data to produce bad intelligence. It can produce bad intelligence from good data that has been given the wrong meaning. The AI didn't fail. It answered the question we gave it. We simply positioned ourselves incorrectly before we started.

The Wrong Place on the Map

The old computing expression is garbage in, garbage out. It remains true, but it doesn't fully describe this problem. The dangerous input isn't always garbage. Sometimes it is perfectly accurate information answering the wrong question. Imagine using a navigation system. You have an excellent map. The destination is correct. The system calculates the optimal route. Every turn is precisely described. There is only one problem. You told the navigation system you were starting from somewhere you aren't. You can follow every instruction perfectly and still fail to arrive where you intended. The map wasn't wrong. The route wasn't wrong. The directions weren't wrong. The starting assumption was wrong. AI can create the same problem at extraordinary scale. Greater analytical power does not automatically correct a mistaken premise. In fact, it can allow us to proceed from that premise faster, more confidently and with considerably greater sophistication. That is why information verification must become more than checking data for errors. We also need to verify meaning. What does this measure actually represent? What assumptions are embedded in it? What information has been excluded? How was it transformed? How is its presentation shaping our interpretation? And is it appropriate for the decision we are about to make? AI gives businesses an extraordinary new ability to analyze information and reason from it. But AI does not eliminate the need to understand our measures. It makes understanding them more important. Before asking AI what the numbers mean, perhaps we should first make sure we know what the numbers mean.