The Agile versus Waterfall argument divides software developers. There are those that believe iterative development to be superior way of developing software, and those that believe up-front planning to be a superior way of developing software. A couple of weeks ago I had a short but passionate argument with one of my co-workers. They are super super smart, and we each stand on opposite sides of the up-front planning divide. My colleague’s argument is that you need to plan in order to execute fast and efficiently at scale. That fast execution at scale requires considerable up-front planning.
This puzzled me greatly. Real Options shows that up-front planning is inferior as an approach because you commit early and destroys lots of options. Lean tells us that planning is a form of Muda or waste. This is particularly interesting because if people smarter than me think this, then real options could be a flawed concept and I break the model… and learn as a result. After much discussion and thought, I think I may understand the problem. The insight comes from thinking about the problem using the Cynefin model for social complexity developed by Dave Snowden.
The Cynefin model consists of four quadrants (snigger – In joke) of “Simple”, “Complicated”, “Complex”, and “Choatic” and a fifth quadrant “Unordered” for when you do not know which of the other quadrants you are in. A “Simple” system is over constrained and has cause/effect behaviour. “Complicated” is the realm of the expert. “Chaos” means there are not enough constraints. “Complex” means there are just enough constraints to create an emergent behaviour which can be nudged and tweaked into a system with desirable behviour. “Complicated” means an expert can reduce the problem to component parts with predictable behaviour. “Complex” means there are feedback loops, agents and limits that create an emergent behaviour that cannot necessarily be predicted. Essentially, the difference between the two can be two can be distilled to “A complex system interacts with its context and evolves as a result, and a complicated system does not interact with its context or evolve.”
If you know in advance what is needed, you are in a complicated system. You can specify and plan in detail to create an optimal delivery at scale (A long lead time and short duration). Hence the space shuttle. Complicated systems struggle to react to changes in context which can cause analysis paralysis. In product development, complicated systems are managed and dominated by experts. The axioms of this paradigm are untested assumptions. This is the realm of early commitment and the slaying of valuable options. As a cautionary note, these systems have the potential to suffer a catastrophic failure if they choose to ignore the context changes.
If you are in a situation where the context is dynamic and an expert cannot predict in advance, you are in a complex system. You cannot specify and plan in detail because the context changes before you can deliver. Development in a complex system consists of several safe to fail experiments followed by fast developments to enhance beneficial developments and dampen non-beneficial developments. In product development, decisions are made by studying data and the impacts of safe to fail experiments and small iterative investments. Products and their context co-evolve. The axioms of this paradigm are hypotheses which are to be challenged and tested. This is the realm where real options are an optimal strategy (Check out Dave’s paper from 1857).
So the answer is quite simple. If your product is in a complicated context, use experts and up-front planning. If you are in a complex context, create options and learn to respond faster than your competition.
A final note. If you think you are complicated context, and you are operating in a market with competition… I’m sorry, you are wrong. You are actually in a complex context.
Despite my best efforts, I now find the Cynefin framework to be a useful tool to understand* and operate** the context within which I work. If you want to find out more about Cynefin, sign up for one of Dave’s course at www.cognitive-edge.com. Oh, and refer to the quadrants as quadrants. He love’s that.
*Phenomenology
**Ontology
The IT Risk Manager 
March 25th, 2014 at 4:08 am
[…] Responding to Change over Following a plan Written by: Chris Matts […]
March 27th, 2014 at 8:58 am
The fifth ‘quadrant’ is disorder not unorder. The furthest edge of Chaos is unordered. I was at a book club discussing the book ‘Switch’ last night and puzzling over why (and I appreciate it being mostly in hindsight) a ‘clearly defined goal’ is such a powerful force, even in a complex, emergent situation, as without it, there is less commitment.
August 12th, 2014 at 6:22 pm
[…] was when reading one of Chris Matts’ blog posts that it struck me that what Reinertsen was describing was the difference between complicated work […]
January 8th, 2015 at 3:01 pm
[…] Responding to change over following a plan – by Chris Matts […]
January 8th, 2015 at 6:03 pm
[…] – by Renee Troughton Customer collaboration over contract negotiation – by Jared Richardson Responding to change over following a plan – by Chris Matts Try out our Virtual Scrum Coach with the Scrum Team Assessment tool – just $500 for a team to get […]
January 9th, 2015 at 1:11 am
[…] What is Agile? In 2001, 17 people got together for a world-changing discussion about software development. They tried to find the common values and principles by which people could do better at the work of software development, which was in a terrible crisis (and still, to some extent is). They were successful in that they created a list of 4 values and 12 principles to guide people trying to find better ways of developing software: the Agile Manifesto was born. Now, nearly 14 years later, Agile software development has become well-known (if not well-practiced) throughout the business world. In fact, the concepts of Agile software development have been extended through to many other fields as diverse as mining, church management, personal time management, and general corporate management. In the process, there has also been a growing recognition of the relationship between Agile values and principles and those of Lean thinking. It is time to think about the concepts behind the values and principles. To acknowledge that the Agile Manifesto (for software development) can be re-stated at a much deeper level. To abstract Agile software development to Agile work in general. This is my goal over a series of essays about the Agile Manifesto. Let’s start with an analysis of the values of the Agile Manifesto in relationship to the concept of “value”. The Agile Manifesto Values In the Agile Manifesto we can read the four values: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan While a few of these already are quite general, let’s dig a bit deeper by starting with the second value, “working software over comprehensive documentation”. What does this value really refer to? Why do we care about software over documentation. Why “working” vs. “comprehensive”? This is where Lean thinking can help us. The notion of “customer value-added” or just value added work is any work that changes the form, fit or function of that which you are delivering and simultaneously is work that a customer would pay for independently of all the other activities and results you may be spending time on. In this specific example of software and documentation, we can try to imagine what it would be like to say to a potential customer, “we can give you documentation for our software, but we won’t be able to give you the actual software itself… but we’d still like to be paid.” I’m sure it will be clear that it would be a very unusual customer who would agree to such a proposal. Thus, we see that the value of software development activities is in producing the software itself, and the documentation is by necessity of secondary importance. But what if we are writing a book of fiction? Surely this is documentation! But, it is not. To make the analogy to the type of documentation mentioned in the Agile Manifesto, we would sell a book not just with the story itself, but also with in-depth instructions on how to use a book, how to read, how to interpret our feelings as we read, etc. And not just that, but we would also provide a set of notes to the publisher about exactly how we wrote the book: the time and place of each paragraph written, our original outlines, our research including much that was thrown away, all our conversations with people as we struggled to sort out various plot, character and setting elements, and possibly even all our edits that we had thrown away. This is the documentation to which, by analogy, the Agile Manifesto is referring. Perhaps now we can look at a connection between the first value and the second value of the Manifesto: that documentation, tools and processes are all much of the same thing. They all belong to the same abstract category, namely, the means used to achieve a particular end. Of course, we all know that both the means and the ends are important, although we may not all agree on their relative importance. Nevertheless, we can probably agree on some extreme outliers that will help us come to the point I wish to make. For example, we can agree that killing someone merely to get to the front of the grocery store line and save a minute or two of time is an extreme case where the end clearly does not justify the means. Likewise, we can also agree that refusing honestly given help out of a desire for independence when it ends with the death of our children by starvation is putting means too much in the fore. Balance is required, therefore. The Agile Manifesto acknowledges this balance by its epilogue to the values, “That is, while there is value in the items on the right, we value the items on the left more.” The other two values of the Agile Manifesto which mention contract negotiation and following a plan are similarly pointing out activities of the means that are non-value added, in Lean terms. The Agile Manifesto, is stating four values, is, quite directly, pointing to those things which in life are fundamentally of value as ends, not just as means. Individuals and interactions, working software, customer collaboration and responding to change, are all valuable. In order to abstract away from software, then, and create a more general statement of the nature of Agility, we need to explore the idea of value. The Idea of Value If we are in business, determining value, while possibly complicated, is not usually too obscure an effort. We look at exchanges of money to see where the “market” agrees there is value. The price of a product or service is only representative of value if someone will actually pay. Therefore, businesses look at return on investment, profit margins and the like to determine value. Similarly, in other types of markets such as the stock market, value is determined by an exchange of money. But underlying this exchange of money is a decision made by an individual human being (or several, or many) to refuse all the other potential uses of that money for the one specific use of making a particular purchase. This choice is based on all sorts of factors. In economics, we talk about these factors mostly in rational selfish terms: what sort of benefit does the purchaser get from the purchase. Factors such as risk, short-term vs. long-term, net present value, trade-offs, etc. certainly can play a role in such decisions. But in business (and in particular marketing and sales), we know that there are also lots of non-rational forces at work in deciding to spend money on a particular something. Value, therefore, has a great deal to do with how a particular person both feels and understands the current proposed “investment” opportunity. Feeling and understanding arise from many specific factors, as discussed, but what can we say generally about feeling and understanding? They are internal states of a person’s mind (or heart, if you like). Those internal states have been the subject of much discussion in the realm of psychology, sociology, economics, and philosophy. But quite simply, those internal states are greatly determined by perception. How a person perceives a situation is the immediate general factor that determines those internal states. Of course, perception is a general term that includes sensory perception, but also the kinds of prejudices we have, the categories into which we place things conceptually, the internal language we use to describe things, and our existing emotional and mental constructs. So, fundamentally, value is perceived depending on all these perceptual factors. The Agile Manifesto authors, therefore, had (and perhaps still have) a perception of value which places individuals and interactions, working software, customer collaboration and responding to change all in a position of more value than those other items. But this perception of value may not be in alignment with other people’s perception of value. Still, we can see already in 13 short years that there is broad, if not universal, agreement on these statements of value. Why is this? Why has Agile become so popular over a relatively sustained length of time with a trajectory that still seems to have it growing in popularity for years to come? Agile values address a deep need in people in the software development discipline and indeed, by analogy, into much of the work world. In my next essay on the Agile Manifesto, we will take a look at some examples of abstracting the Agile Manifesto to non-software work. If you know of attempts to do this, please let me know in the comments!!! Some links to commentary on the values of the Agile Manifesto while you wait for my next essay instalment!… Individuals and interactions over processes and tools – by Mark Layton Working software over comprehensive documentation – by Renee Troughton Customer collaboration over contract negotiation – by Jared Richardson Responding to change over following a plan – by Chris Matts […]