Author Archives: theitriskmanager

About theitriskmanager

Currently an “engineering performance coach” because “transformation” and “Agile” are now toxic. In the past, “Transformation lead”, “Agile Coach”, “Programme Manager”, “Project Manager”, “Business Analyst”, and “Developer”. Did some stuff with the Agile Community. Put the “Given” into “Given-When-Then”. Discovered “Real Options”

Automated Test Coverage as a goal is at best, misguided

Setting Automated Test Coverage as a goal is at best, misguided. Automated test coverage is useful as a strategy or as a diagnostic metric, however using it as a goal is idiotic and will lead to waste and the wrong behaviour.

hand-697264_1280

For any IT system, there are three options for testing:

  1. Automated tests
  2. Manual tests
  3. No tests

Lets pop the why stack on automated tests. Automated tests are faster and more reliable than manual tests. Automated and Manual tests are normally safer than no testing. So the reasons for automated tests are:

  • Reduced lead time.
  • Reduced variability in lead time.
  • Lower probability of a production incident.

Our goal should be to improve one of these metrics, normally reduce lead time. Lead time and automated test coverage are correlated. If you attempt to reduce lead time, one of the strategies you are likely to apply is to increase automated test coverage. As such automated test coverage is an excellent diagnostic metric to help the team identify ways to reduce their lead time.

There is not a causal relationship between automated test coverage and lead time. Increasing automated test coverage does not automatically reduce lead time. Many years ago I worked on system with no automated test coverage. Management imposed a 100% test coverage goal for all systems. Everyone on the project stopped working on anything else and spent a few days writing tests. As the business analyst I was given a list of classes and told how to write standards tests for each method to ensure the test coverage tool would register us as meeting our 100% target. We achieved 100% automated test coverage but no improvement in lead time or anything. The activity generated no benefit to the organisation, it was pure waste.

If you set reducing lead time as a goal, you will likely see an increase in automated test coverage. If you set increased automated test coverage, it is possible you will see no benefit.


Investing with Cynefin: Disorder

Disorder is the fifth quadrant in the Cynefin model. Disorder is where there is no clarity about which of the other domains should apply.

“Here, multiple perspectives jostle for prominence, factional leaders argue with one another, and cacophony rules”.

hippo-3645651_1920

The solution is “obvious”, bring the leaders together to perform a “complicated” ritual that reveals the “complex” nature of the problem and hope that the personalities involved do not turn the conversation into something “chaotic”.

A technique like four corners contextualisation can be used to facilitate a conversation between decision makers. The discussion will normally reveal that the problem is one of granularity. Decision making is a classic example of something that is often in the domain of disorder. However, when we break it down into a lower level of granularity, we discover that it falls in the other four domains.

  • The output, an ordered list is obvious.
  • The process such as cost of delay is complicated.
  • The interaction between the investment options, the available resources and the participants are complex.
  • The behaviour of warring factions is chaotic.

By moving to a lower level of granulation, the domains become apparent.

Investments that remain in the disorder quadrant indicate a dysfunctional decision making group. Often a hippo will force these items into one of the other domains. Disorder is often a symptom of a group stuck in “storming” that is not coming together to communicate.


Investing with Cynefin: Complicated

The complicated domain is the realm of strategic superiority. Organisations know with certainty how customers will behave, however it is not common knowledge. This is where organisations should focus their strategic investment.

star-2496357_1280

The complicated domain is where organisations should be making larger investments as this is where they have competitive advantage. The only place where organisations should be making larger investments is where they are forced to due to regulatory dictate (Obvious) or to resist the vertiginous draw of the cliff (The Cliff). The complicated domain is where constrained resources should normally be deployed.

The complex and complicated domains are not binary in nature but rather “linear” with 0% certainty at one extreme and 100% at the other. As such, the investments are best managed using the Kelly criterion. The nature of the experiments change. Whereas in the complex domain, the experiments relate to understanding needs, in the complicated domain, the eperiments relate to the scope of the needs.

For organisations, the danger with the complicated domain is that too many investments are made in it. Either because they are classified incorrectly due to perverse cultural incentives or because the organisation is utterly risk averse. One is reminded of the risk averse anthem “No one ever got sacked for investing in the complicated domain. buying IBM.”. The real message being that perhaps some people should have been sacked for failing to think for themselves.

In summary, investing in the complicated domain is the easy option. Therefore the investment decision process should make it difficult to do so.


Investing with Cynefin: The cliff

In the Cynefin framework there is a cliff between the Obvious and Chaotic domains. Systems fall down this from the ordered domain down into the Chaotic domain. This is eloquently summarised by Mark Twain…

It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.

action-1838330_1920

Retail businesses on the high street are currently thrashing around at the bottom of this cliff. After years of pulling levers to control shoppers and extract profits, the bricks and mortar retailers now resemble Dr Who randomly playing with the controls of the Tardis as it bounces around reality. The reason is that the internet has changed the fitness landscape of business and new apex predators have emerged that are disrupting existing players.

When organisations find themselves inside the OODA loop of an Apex Predator in an unfamiliar fitness landscape, they have to act, sense and respond. As a result, investments in the chaotic domain may result in disrupting the investment process in all of the other domains. Although investments in the chaotic domain should ideally be small, there are occasions when all the resources of the organisation will need to be focused on them.

Ideally organisations should avoid the cliff all together, and if necessary they will need to invest all of their resources to keep themselves away from it. Organisations can use metrics to detect the presence of the cliff. In particular they can use churn metrics. An organisation would typically have three sets of customer metrics…. Number of customers, customer activity and customer revenue. The trend is more important than the actual value for investment decision making.

Churn

In the example above the metrics may initially appear healthy. Every week sees a 10% increase (Green). However looking at the churn number indicates a problem occurred in week 4. At this point, the organisation needs to act as it heads towards the cliff / lands at the bottom of it. The action required is normally to gather information. Why has churn just jumped up? Which customer need is not being met, or possibly which customer need is being better met by a competitor’s product or service? The research will either involve data analysis or user experience research (to understand user needs/jobs to be done).

Many organisations wallowing at the bottom of the cliff have no data analysis or user experience research capability. For these organisations, the ‘act’ is simple, to acquire these capabilities.

Modern organisations wishing to avoid the cliff need to invest in data analysis and user experience research capabilities before they find themselves being disrupted by another organisation’s OODA loop.


Investing with Cynefin: Complex

For investments, the complex domain may be defined by the strategy used to manage it “Multiple hypotheses tested using safe to fail experiments”. The complex domain by definition contains uncertainty (risk) and is about acquiring knowledge. Investments the complex domain should be the smallest safe to fail experiment that test a hypothesis. Organisations should not be making investments that are large or that are not safe to fail.

sunset-2295669_1920

Investments involving technology have three categories of risk¹ :

  • Delivery risk.
  • Business Case Risk.
  • Risk of damaging the existing business model.

The Kelly criterion can be used to help understand the maximum size of investment in the Complex domain. Fundamentally investments in the complex domain are about reducing risk (uncertainty) by acquiring knowledge (certainty). The first two categories of risk are about failing to achieve an upside, they are not about protecting against a down side. Managing the risk of failing to achieve an upside is done by ensuring that individual investments are not so large that they damage the portfolio. Hence the Kelly criterion can be used. Whilst Kelly can assist with the first two categories of risk, alternate strategies are needed to protect against a down side.

Managing the risk of damage to the existing business model.

Protecting against down side is about managing the risk of unintended consequences. If a consequence is known, then a specific option should be created to ensure that the investment is safe to fail. Therefore ensuring investments are safe to fail, requires the investor to have options to detect problems and return the system to safety. This means the following:

  1. Effective monitoring is required to detect unintended consequences.
  2. Options to return the system to safety.
  3. Failure containment.

Failure to monitor for unintended consequences is an abdication of responsibility and normally indicates a risk averse culture dominated by Hippos. The Hippos either accept of ignore risks that might occur.

Time is the key element of options. If the time the system can survive is less than the time it takes to return the system to safety, more options are required before the investment should be considered safe to fail.

Finally, one of the key differentiators between contemporary organisations like and Google, Facebook and Netflix and traditional organisations is that contemporary organisations manage risk rather than ignore it. They create failure containment.Contemporary organisations roll out investments to customers gradually. They test hypotheses to ensure that not only does the investment work, but also the customers have the anticipated, or at least a beneficial, behaviour change. The idea that you would roll out an investment to 100% of your customer all at once is the equivalent of putting all of your money on “red” at the roulette table… You are not investing, you are gambling. You might get away with it a few times but eventually you will do a “Knight Capital”.

The Complex domain is fundamentally about risk management. As such, it is the domain where Real Options are most effective.

1-Original article by Steve Freeman and Chris Matts. Published in Agile Times, 2005


Investing with Cynefin: Obvious

The counter-intuitive aspect of investments in the obvious domain is that organisations should seek to minimise investment in this domain. As Dave Snowden once pointed out, organisations have no competitive advantage in the obvious domain. Investments in the obvious domain are often as a result of constraints imposed by regulatory bodies.

screw-1924174_1920

Many organisations have a special categorisation for investments “Regulatory and Mandatory”. These investments are often referred to as

“Must be done!”

A more intelligent description would be:

“Must be done if the organisation wishes to remain in that particular business.”

The reframing is very important. In the late naughties, an investment bank faced with expensive regulatory investments decided to sell its commodities business. This freed up capital that could be invested to make its other business lines stronger.

A common mistake when investing in regulatory and mandatory investments to to try and be the best, to attempt to invest and excel. Using Niel Nickolaisen’s purpose alignment model (Check out Kent McDonald’s excellent description here along with other useful tools), regulatory investments fall in the parity quadrant. Investors should seek to be “good enough” but not invest and excel. This often means implementing a third party solution if an appropriate one is available. Where an appropriate third party solution is not available, the solution should be architected in such a manner that it is easy to migrate to one when it is available.

The strategy for regulatory investments is to minimise total cost of ownership with a “good enough” solution. Unfortunately organisations often misinterpret this as “Implement with your cheapest resources” which is a path to failure and excessive costs. Given that regulations in a market normally increase and are rarely removed, the organisation should consider the long term implications of any solution. This means that regulatory investments should be implemented using eXtreme programming techniques that support safe, rapid and cheap modification in the future.

Realistically, the only way to turn regulatory investments into strategic investments is to deliver a solution to the regulator before your competitors. That way, your organisation can influence the regulators and disrupt any competitors using traditional techniques.

In summary, even though the investment may be obvious, the solution may require careful thought.


Investing with Cynefin: Chaos

To properly understand the Chaos domain in Cynefin, you must consider it from the perspective of the Complex Domain. In the Complex Domain there is enough information for us to form multiple hypotheses. By defintion, in the Chaos Domain, there is not enough information for us to even create a hypothesis. We need to act in order to form hypotheses. Hence:

  • Act
  • Sense
  • Respond

Investments in the Chaos domain should fundamentally be about gaining information.

glasses-1246611_1920

Novelty often occurs in the Chaos domain because there is a need to act but no known solution. As a result, it is easy to see that innovation occurs in the Chaos domain as no hypotheses exist to guide the investment. Many investments in the Chaos domain are often pure bets where by definition, the pay off is unknowable.

However there is a form of investment in the Chaos domain that is not a pure bet. A form of investment that is often scorned and overlooked… An investment in insight and measurement.

Many organisations invest millions, or even billions in transformation efforts to improve themselves. These organisations will spend a fortune on “consultants” and “thought leaders” but struggle to find funding to demonstrate whether the changes are successful. The reason is that it is impossible to put “accurate” cost saving or revenue estimates on being able to see. Success is often a matter of an executive’s opinion, rather than being based on data. An organisation that cannot measure whether it is successful with an investment will continue to make bad investments, and will fail to learn how to make good ones. The more accurate the accurate the measurement, the more efficient the learning.

Investment in the measurement of the organisation should be overseen by executives to ensure that it happens. An absence of transparency is a failing of the culture, and hence a failing of executives who have a responsibility to investors.

Before investing in a transformation, the organisation should first invest in the measure that will prove the success or otherwise of the transformation. For an Agile transformation, that means investing the measurement of lead time (weighted lead time). This measurement is most easily done on historic data rather than waiting for new data. All that is needed is to link existing data to value rather than functionality. However the value of reorganising existing data is often considered waste and so organisations continue without insight.

Executives who engage on transformations without measurement of lead time time are like drivers who close their eyes whenever they get behind the wheel of a car.


Enabling Constraints and Hippos

One of the most profound insights on enabling constraints came from Marc Burgauer… “Enabling constraints cannot form if a dominant hippo is involved in making the decisions”.

rooster-1001892_1920

Enabling constraints form agents into a higher order systems. The higher order system provides feedback to the agents to constrain their behaviour and stabilise the enabling constraint.

Part of the process of enabling constraints is the micro-conflicts where the agents give and take in order to align with each other. This will only happen between “peers”. “Peers” means that one of the agents is not a dominant Hippo around whom all the other agents attempt to align. The dominant hippo might be considered the “Apex Predator” in apex predator theory. The apex hippo can dispatch any member of the group without any fear of reprisals. This means that the dominant hippo always wins the prisoner’s dilemma regardless of the outcome. The only way the apex hippo will fail is if the fitness landscape changes, i.e. from a change in the context, or an outside context threat (i.e. Another hippo from the outcome).

Teams and communities that trust each and work in aligned manner do not emerge with an apex hippo present. Instead, identity is formed based on the relationship with the dominant hippo. The goal of each member of the team is to align more closely with the dominant hippo than their colleagues. The goal is to maintain the pecking order. Relationships with other members of the team are incidental and unimportant.

At Skype, we had a apex hippo. Andrew Sinclair was the executive in charge of product. Andrew intuitively understood this and refused to engage in the decision making process. Instead, Andrew used his status and influence to act as a guardian of the process, frequently reminding participants of their responsibilities and the rules of the game, and keeping the group focused on the goal.

The Extreme Tuesday Club in London is the most successful community I’ve been a member of . Simply identifying yourself as a member lead to an immediate trusted relationship with other members. XTC did not have an apex hippo, however XTC had many many leaders to the point where even identifying the leadership was difficult. Conflict was about the only constant at XTC. Conflict and that it always happened on a Tuesday.

I have observed other groups based around an individual or a small group of individuals. Those individuals feel that the community is theirs. None of them evolved into a community where members would bond with each other purely on the basis they were interested in the same thing. More often than not these communities have leaders who are anointed by the apex hippo. They have a revolving door of new members joining as old members leave. They achieve little other than to act as a marketing channel for the apex hippo.

So if you want to build a team, or build a community around an idea, you will need to create an enabling constraint, stimulate micro-conflict, and prevent conflict avoidance.

However,

First kill the apex hippo… especially if it is yourself.


Enabling Constraints and Micro-Conflicts

Luke Hohmann’s innovation games include two classic enabling constraints, 20/20 Vision and Buy-A-Feature. Examining these and other constraints reveals that an important part of enabling constraints may be micro-conflicts.

cat-1234947_1920

20/20 Vision

20/20 vision is a simple and effective innovation game for prioritising a list of items. A number of people come together either physically or on-line using Luke’s Hohmann’s Weave product. The group start with a list of items and randomly pick one. They then randomly pick another and compare its priority to the first item. It can be higher or lower but never the same priority. The third item is then compared to first two. For each, the group decide if the new item is a higher or lower priority. This process is completed for as much of the list as the group wants to prioritise.

The simple rules of the game create a constraint.

  • The list must be strictly ordered.
  • Compare one item at a time.

The constraint ensures conflict between the stakeholders. They have to engage with each other and discuss the relative priority of each pair of items. It is not possible for an individual to order the list without engaging with the other stakeholders.

In order to win each contest when comparing two items, the stakeholders need to articulate the value of the item and why they need it. By doing this they reveal their intent.

Buy a Feature

Buy a feature is another of Luke’s Innovation Games. The game is played by a small group in person or online. A list of features is provided, each with its estimated cost. Each player is given an equal amount of money. The total money available will only cover a portion of the features. Many of the features require the money from more than one player. The players have to build their ideal backlog, convincing the other players to invest in the features they value the most.

As with 20/20 Vision, the constraints of the game ensures conflict between the players, and requires them to reveal their intent in order to convince the other players.

Conflict and Collaboration

When groups come together they are in the forming stage. They are careful to ensure that their concerns do not overlap with others until they know them better. They try not to step on each other’s toes as they do not want to be judged harshly by the rest of the group.

Forming

As individuals gain confidence, they grow their area of concern. As a result, the concerns of one or more individuals overlap. This leads to conflict. Through the conflict, they learn how to resolve issues between them.

A number of things can happen:

  • The group grow to become a team who collaborate with each other
  • The group gets stuck in conflict
  • The group avoids all conflict

Groups that become teams engage in conflict when their concerns overlap.

Conflict

They will argue over small things at first and then bigger things. When they become a team, they do not necessarily expect everyone to adopt the same values and beliefs. In fact, they grow to value and respect the values and beliefs of the other people in the team. In other words they validate the identity of the other team members in the team. They value the differences to themselves and the diversity of the team.

A healthy team engages in many micro-conflicts as they give and take on the road to becoming a team. For example, they might win the battle to adopt “Given When Then” but give up on the adoption of “Mock Objects”. “Giving in” is an important part of the team building process. It is the “giving in” that forms their new identity as part of the team. It is the price they pay for their new identity.

A healthy team engages in conflict all the while. The conflict focuses on the ideas and things with an understanding that good things will come from the discussion. A healthy team does not challenge the identity of the people in the team. Compare “Boris is a pain in the neck, he challenges everything we suggest which makes every discussion harder, and we are slower for it.” and “Sarah is brilliant, she challenges everything we suggest which makes every discussion harder, and we are get awesome outcomes as a result.”

Most people dislike conflict, especially over the long term. If the group cannot resolve their differences, they are more likely to avoid conflict by avoiding each other. If the group does engage in direct conflict for a prolonged period, the whole group will fail. Failing groups tend to draw more people in which makes failure more visible to the wider organisation. A public failure of the entire group is normally enough to a catalyst to tip them into norming. ( As an aside, I have found Dan Mezick’s triad concept to be an effective structure to help groups at this point ). The norming phase is early collaboration where people often over share for fear of causing another group failure, or they gradually share increasing amounts of information. In both cases, they get to a point where trust is established and they understand what other people want to be informed about.

If the group avoids conflict, the boundaries are carefully negotiated to prevent conflict. I call this the cold war.

Avoiding Conflict

Skirmishes occur at the borders but sharing and communication are contractually agreed. These contracts often involves one or both parties telling the other the things they can and cannot do. Consider the relationship between Business Analysts and Developers, and Developers and IT Operations. Service Level Agreements are a clear indicator of a cold war relationship between two groups. When these conflicts fail, they can be quite unpleasant. These conflicts are like the Cuban Missile Crisis or Vietnam where the two main powers (e.g. Executives) do not engage directly but rather engage in a proxy war. For the people directly involved, the impact can be devastating.

Avoid conflict guarantees that the group will never evolve into a team, avoiding conflict perpetuates conflict and prevents collaboration from emerging.

The Strategy of Conflict

The strategy of conflict is a game theory strategy developed for the United States to defeat the Soviet Union during the Cold War. It is a very effective strategy for “winning” when you are in conflict with others.

It has several strategies for winning:

  1. Hide your intent.
  2. Withhold information.
  3. Do not communicate with your opposition.
  4. Do not allow competitors to have access to your decision maker.

Both 20/20 vision and Buy a feature constrain members of the group to do the opposite to one, two and three. Both require people to share their intent, why they want an item, and how it will help them. To improve the chances of an item “winning”, both require people to share all the information they have. Both force communication.

I am aware of examples where the decision makers come together and form the enabling constraint. Perceiving that the situation is in control, they send a proxy on their behalf. Where the proxy is not fully authorised to speak on behalf of the decision maker, and the decision maker is not bound by the commitments made by the proxy, the enabling constraint falls apart. If a stakeholder is unable to partipate in an enabling constraint, they must empower their proxy and commit the decisions made.

Enabling constraints need to be constructed in such a way as to prevent members from avoiding conflict, and create a dispositionality towards collaboration.

Micro-Conflicts

My hypothesis is that the micro-conflicts evident in 20/20 Vision and Buy a Feature are an important but often overlooked part of the process of forming an enabling constraint.

An enabling constraint creates micro-conflicts that dispose people to “give up” and invest in the team. This investment means they value the team, and they value the identity they acquire by becoming a member.

Budget Games

Since 2010, Luke Hohmann has been helping the Mayor of San Jose engage with the population about its budget. Budget Games, a large scale implementation of buy a feature, has been run every year since.

Whilst the purpose of the exercise was to order the backlog (prioritise the budget), one of the impacts has been that Budget Games has also built and strengthened the community in San Jose.

The community has been formed as much by years of micro-conflicts where some people give, and some people receive.


Capacity Planning and Cost of Delay

Effective prioritisation has two outcomes, it creates an ordered backlog that can be used to align an organisation’s activity, and it forms decision makers into a team. The first outcome can be considered tactical and the second strategic. Arguably the second is more valuable than the first as it will result in optimising the outcomes for the organisation rather than each decision maker attempting to optimise their own outcome. When each decision maker attempts to optimise their own outcome, the outcome for the organisation is often sub-optimal.

formation-508038_1920

Now that we  understand that there are two outcomes, we can ensure that both occur. All to often we will deliver the tactical ordered backlog but fail to form the decision makers into a team. We need to ensure that we form the decision makers into a team AND we order the backlog.

An enabling constraint is one that creates a higher order system that provides feedback to the agents in the lower order system and constrains their behaviour. In concrete terms, an enabling constraint takes a group of individuals and constrains them in some way to form them into a team. Becoming a member of that team feeds back into their identity and constrains the way that they behave. For example, membership of the team may constrain members to consider the needs of the organisation above their own needs, goals or targets.

A decision making tool helps a team to order the backlog. It provides a common frame of reference to enable comparison of disparate items.

Sometimes it is possible that a single tool can achieve both outcomes. At Maersk, Cost of Delay was (apparently*) used not only to order the backlog, but also the mechanism to bring the stakeholders together. Cost of Delay acted as an enabling constraint that aligned the decision makers. In the context where all investments have to be expressed in financial terms, Cost of Delay can act as an enabling constraint.

In some contexts, it is useful to separate the enabling constraint that forms the team from the decision making mechanism. This allows you to change the decision making mechanism without losing the team. If the enabling constraint and decision making mechanism are too deeply entwined, failure of the decision making mechanism may cause the enabling constraint to fail. For example:

  • An economic decision making mechanism is being used and an important investment in the complex or chaos domain arises that causes the entire process to fail.
  • The team has ordered the backlog and a Hippo announces changes to portfolio that they cannot share due to regulatory reasons. (in this situation, the team should ideally be subjected to an NDA and introduced to the privileged information but that is not always possible or desirable).

Sometimes it may be possible to establish an enabling constraint but the organisation cannot establish an economic decision making framework. In contemporary organisations, economic metrics are seen as lagging indicators of success, and other metrics are seen as equally important. These other metrics might be number of customer, activity or lead time. In such an organisation, it might be impossible to land cost of delay as a decision making mechanism. However, establishing the enabling constraint may make it easier to establish an economic decision making framework at a later date… if necessary.

In other words, the team of decision makers need to be able to choose the most appropriate mechanism for ordering the backlog without feeling they are abandoning the process that forms the team.

Final Thought

Only members of the Community of Solutions insist that a particular solution is the best in all contexts, for example insisting that we use Capacity Planning OR Cost of Delay everywhere. Members of the Community of Needs are more comfortable with using different solutions to satisfy a need, for example using Capacity Planning AND Cost of Delay.

Capacity Planning and Cost of Delay are complimentary concepts. Understanding them as separate and independent tools allows us to create more resilient portfolio processes.

* This is a second hand anecdote.