For any Executive thinking of introducing Metrics, they need to understand that there are three level of Maturity:

• Step 1: Map Investment to Metrics.
• Step 2: Use Metrics to demonstrate Success.
• Step 3: Use Metrics to assist Decision Making.

Demonstrating success is the next step after the organisation achieves alignment using Metrics.

Consider the hot air balloon captain who got lost in the fog.

• “Where am I?” they shout.
• “You are in a hot air balloon!” comes the reply from the ground.
• “Are you an Executive without Metrics?” asks the Air Balloon Captain.
• “How did you know?”
• “Because your answer was one hundred percent accurate and utterly useless.”

If you do not have metrics, your decision as to whether your investment yielded a success is based on opinion and not facts.

To quote Jim Barksdale, former CEO of Netscape:

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

In other words, if you do not have data to demonstrate the success of an investment, the CEO, an executive or steering committee decides whether it was or not. So why is that such a terrible situation?

When an organisation makes an investment, it is either a success or it is a failure. If it is a failure, the organisation learns (normally about its customers) so that its subsequent investments have a greater chance of success. If an executive decides that the investment is a success when it is a failure, then the organisation is denied the opportunity to learn and it may continue to make poor investments. Executives who declare success based their opinion rather than data are cheating the organisations that they are morally obliged to serve and protect. They are cheating the organisation out of an opportunity to learn.

This leads us to the second level of metric maturity…

Level 2: Demonstrate Success

No one would question that in general Start Ups are much more successful than traditional organisations at innovating and learning about their customers. The reason is simple, they use data extensively to determine whether their hypotheses about their customers are right or not.

The first step to demonstrating success is to establish a baseline. The baseline is the “value” of the metric before the investment.

The baseline…

1. …should be for a metric that is either a sub-set of the business value metric, or another metric where the executive believes that they have a hypothesis that moving the metric will move the business value metric.
2. …only has to be accurate enough to prove whether the investment was a success or not.
3. …needs to be calculated using the same method as the result of the investment.

All three of these points will be covered in more detail in subsequent posts.

As an aside, it is easy to spot a culture where the executive’s opinion decides success because Product Owners do not care about establishing a useful baseline.

Once you have a useful baseline, the rest of the process is simple:

1. Articulate your hypothesis.
2. Construct an experiment to test the hypothesis.
3. Run the experiment.
4. Measure the metric and compare the results with the baseline.

Note: The baseline may be determined at any point prior to running the experiment.

There are three possible outcomes:

1. The investment is a success. Our hypothesis is correct, we understand something about the customer and we have successfully created business value.
2. The investment is a failure. Our hypothesis is incorrect. However we understand something about the customer and we have a better chance of creating business value in the future.
3. The results are inconclusive. It was a bad experiment and we need to construct a better one to test the hypothesis. We may or may not have learned something about the customer.

Whether success is demonstrated using a metric or the opinion of an executive is a cultural matter. It is purely the responsibility of the executives as it is their decision whether to allow investments to take place that do not demonstrate success using metrics. If an executive wants to help their organisation succeed and insist on learning about the customer, they should simply fail any investment that does not use data to demonstrate success.

At Skype we had a metric to measure this… “The percentage of the investment portfolio that could be demonstrated using metrics”. More on that later.

The next blog post will look at how to “Use Metrics to assist Decision Making”.

Photo credit: https://pixabay.com/en/users/Pexels-2286921/