The Agile Manifesto
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
- Individuals and interactions - over processes and tools
- Working software - over comprehensive documentation
- Customer collaboration - over contract negotiation
- Responding to change - over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Over the next 20 years, those 68 words would come to revolutionise not just their discipline of software development, but business in general. It is hard to find a sector today that doesn’t profess to be ‘doing Agile’ in some form or another. Not just for software products, but for service design, in the public sector, HR processes and organisational transformations.
But, whilst it is easy to find examples of organisations using Agile, a quick search also throws up lots of criticism of the approach and many who have struggled or failed to succeed with it.
Putting Agile into action
This is even true of teams who are trained and experienced. The 2020 Version One State of Agile Survey polled over 40,000 Agile executives, practitioners and consultants; only 16% said they had a high level of agile maturity.
Why is that the case? On the face of it, the agile values and the 12 principles that followed them are simple to understand and the dominant agile approach (used by 85% of teams), Scrum, is described in a guide of just 19 pages.
This is the paradox of Agile - how can something that is seemingly so simple, be so hard to do well?
This paradox can be explained by understanding a number of contributing factors:
- Agile is typically presented as simple to master.
- The Dunning-Kruger Effect explains why it is easy to think you know more than you do.
- People are complex and unpredictable and don’t like change.
- Interventions to try to help often undermine the core values and principles.
The simplicity of Agile is, in many ways, its downfall. Coupled with the Dunning-Kruger Effect, it seduces people into believing that they understand it and can apply it.
Simplicity itself - in theory
After all, it’s just a few phrases; you can read the Scrum Guide in half an hour and be certified as a master in just two days. Agile courses are marketed like many other skills courses and people assume that completing the course will mean they can immediately get results.
This is compounded by the fact that Agile is often presented as a set of rules, practices and techniques that can be learned and followed. We are used to other skills that are very deterministic - when we follow the rules correctly, we will get predictable results. So, it’s easy to see why people assume Agile is the same.
But in reality, very few people succeed straight away and even highly experienced practitioners don’t find it easy. In fact, to achieve the Scrum Alliance’s highest levels of certification (CEC and CTC), the examiners will expect to hear stories of where things have gone wrong and candidates with perfect records won’t succeed because they won’t be believed!
Even the Scrum Guide knows it is hard - the first page states that Scrum is ‘Simple to understand; difficult to master’. But, perhaps because it’s on the first page, it’s not the part of Scrum that people remember and it’s easy to assume that you know it all when you pass the exam and collect your Scrum Master certificate.
The Dunning-Kruger Effect
In 1999, psychologists Dr David Dunning and Dr Justin Kruger described a cognitive bias known as the Dunning-Kruger Effect. They showed that people display Illusory Superiority – We judge ourselves as better than others to a degree that violates the laws of maths.
For example, in one study, 42% of software engineers at one company all judged themselves as being in the top 5%. This leads people to form wrong conclusions or make poor judgements because they rate their level of competence as being much higher than it is.
This effect is most pronounced in relatively inexperienced people and results in a double impact - not only does their inexperience lead to poor decisions, but their ignorance of this prevents them learning from those mistakes.
Dunning and Kruger describe this visually with two axes - actual wisdom (made up from experience and knowledge) and confidence in your ability. As understanding grows early in your development of a skill, confidence multiplies. So quickly in fact, that it’s tempting to believe that you are already an expert.
If your wisdom continues to grow, however, it becomes clearer that there is a lot to learn and your confidence in your ability crashes. Even though you know more than you did, the realisation that there is a lot you don’t understand dents your confidence. This summit - where you don’t know a lot but think you do - is called Mount Stupid.
Fig 1. Dunning-Kruger Effect diagram in detail
Simpler version
Fig 2. Dunning-Kruger Effect diagram - simplified
In Agile terms, taking a Certified Scrum Master course can be sufficient to reach the summit of Mount Stupid; it can be a very pleasant place to be. You are full of confidence in your abilities and ready to take on the world!
If you take the opportunity to develop further wisdom, it quickly becomes apparent that you have a lot still to learn. That realisation can be quite painful and it isn’t always easy to see how you can progress.
However, Dunning and Kruger showed that many people don’t progress beyond that peak. They can make poor decisions, give poor advice and continue to do so because they don’t consider that they could be wrong.
So, if Agile appears simple, yet learning all about it in some of the short courses on offer, isn’t enough to become an expert, what can you do?
One answer is to look at why Agile is hard. One of the problems Agile faces is the very problem that it was conceived to address - the uncertainty of today’s world.
Traditional planning approaches assume that the problem space is relatively stable and predictable and that putting enough effort into the analysis and planning will result in a successful outcome. Agile specifically assumes the opposite - it assumes that there will be change and that the future beyond a few days or weeks is an impossible thing to predict.
There is always uncertainty
Dealing with this uncertainty means that agile teams can’t deal in absolutes and that decisions and judgments are much more nuanced and a factor of the people involved. This uncertainly isn’t just within the team, it’s also with the customers and users.
Stating requirements usually means guessing and trying to predict their needs in the future. These decisions are very people-centric and people being inherently complex and unpredictable adds additional layers of uncertainty and complexity to already challenging problems.
People are resistant to change. They have got to where they are by following a particular path, adhering to particular values and accepting particular truths. When organisations start to adopt Agile, many of those historic truths are challenged. The right-hand side of the Agile Manifesto is a very comfortable place for a lot of people. It is where a lot of traditional management values originate and those values have often served people well over the years.
A clash of cultures
This clash of cultures is a reason that many organisations struggle to adopt agile approaches. Even when teams embody an agile mindset and values. The same is not always true of the management, leadership and governance around them. Agile is more than adopting Scrum’s odd language and unusual events. It’s about a fundamental shift in the things we value throughout the organisation. And because of our natural resistance to change, this can be a hard thing to overcome.
These challenges are not new or revolutionary and there have been many attempts to address them. Often, these changes focus on how to make the differences more palatable to those who are not used to them. Teams and organisations make local optimisations to help it fit in with other processes or find jobs for staff who don’t fit neatly into agile roles. They attempt to make the work more predictable, try to forecast over longer periods or scale to larger teams and organisational structures.
Reframing the problems
Encompassing frameworks have been built around agile teams to address this - things like SAFe and AgilePM are particularly popular. These typically try to help by adding guidance and structure to the agile approaches. This can have one of two effects:
- They can provide guard rails and confidence to agile teams that help them make decisions while upholding the agile values and principles.
- But they can also serve as a rulebook and set of instructions that must be followed.
And for many people, especially those whose history and previous success have conditioned them to seek stability and certainty, that second effect is the one that dominates.
When you consider these four factors, it is easy to see why agile teams struggle, even for experienced and confident teams. But for new, inexperienced or junior teams, it’s even harder. We ask them to follow an agile approach, that they assume is easy.
When they have the training, when they are confident they know enough. Then they interact with people who change their minds, aren’t certain of the future and don’t behave predictably. And we expect them to deliver in a predictable manner. No wonder they struggle.
And that’s before we even think of scaling Agile, which compounds this problem in multiple dimensions.
To find the answer, let’s go back to basics
We have talked a lot about the Agile Manifesto and its left and right sides. Let’s examine it in a bit more detail: The Manifesto says that we have come to value:
Individuals and interactions over processes and tools.
This acknowledges that most complex problems are complex because they involve people. Focusing on people and how we interact will help us navigate that complexity better than trying to predict it in advance and codify it into a process or a tool.
When we value the right more, we are assuming that processes and tools that are optimised to help solve previous problems will also be the best choice for new problems. In today’s volatile and complex world, that’s quite a bold assumption to make.
Working software over comprehensive documentation.
The keyword here is ‘working’ . Whether the solution is a software product or not is irrelevant. What matters is that the sooner we get something to the customer that helps to solve their actual problem, the sooner we can get feedback that we can act upon. It’s not until the user actually uses the product that they can be certain they were right about the problem and the team can be confident they were right about the solution. Earlier delivery also means earlier value for the business.
When we value documentation more, we are making assumptions that the documents are correct and that assumes that everyone who contributed to them was absolutely certain on the facts. That’s rarely true in today’s knowledge work. Testing with working versions can help validate the documents and ensure they are correct.
Customer collaboration over contract negotiation.
Contract negotiation implies an end point that is reached before work begins. The contract may be a commercial one, but it is also often an internal agreement, such as a requirements specification or a product roadmap. It implies that the role of the customer ends with that agreement and that they aren’t permitted to change their mind.
However, when the problem is complex with inherent uncertainty, anything that locks out change will result in a solution that is at least partly wrong and perhaps badly wrong. We need to constantly develop our understanding of the problem and solution and we cannot do that without the customer’s involvement.
Responding to change over following a plan.
This one really sums up the difference between Agile and traditional delivery approaches. In Agile, we expect change, even embrace it. In many other approaches, we try to prevent or control change. This is true, not just in product development, but other fields such as performance management, strategy development or organisational change. It is especially true in project management, where it is critical to being able to deliver predictable outcomes.
When we value following a plan, we assume the plan is correct and that what we thought was true at the start remains true throughout the period. That’s quite a brave bet for many organisations today to make.
So how can we pin-down Agile?
As we have discussed, the seemingly simple Agile Manifesto can be subverted and misunderstood in a multitude of ways. One of the simplest ways to identify if this is happening is to go back to basics: look for examples of where the team or the organisation is valuing things on the right-hand side over things on the left. When this happens, it will be harder to deliver value and more likely that the team will struggle or fail.
To understand what an organisation values, don’t just look at what it says it values, but what it shows that it values. What does it measure? What behaviours are rewarded in its people? Where does it invest, or not invest? How does it treat its people?
It can sometimes be hard to objectively look at teams to see where there are problems. A common solution is to use internal or external Agile coaches. They work with teams and leaders across organisations to help them identify the challenges and systemic problems. They help organisations make thoughtful, considered optimisations and customisations. They do this by coaching, helping the team decide how to proceed while being conscious of the Agile values and principles.
In conclusion
Agile is often seen as a simple treatment that should deliver easy value to organisations that adopt it. It is easy to describe and people can get started just by reading the Scrum Guide or browsing the SAFe or LeSS web pages.
If you want to be an expert, there are lots of short courses where you can become certified. But this apparent simplicity masks a multitude of complex ways that can lead to agile teams failing to deliver value.
One of the biggest culprits is the gulf between the agile mindset required to succeed, described on the left-hand side of the Agile Manifesto and that described by the right-hand side. Unfortunately, this right-hand side mindset is the one that overwhelmingly dominates traditional solution delivery processes and organisational management.
To help organisations adopt an agile mindset or help agile teams to deliver, it is important to recognise this and help shift the mindset towards one that values the left-hand side of the manifesto over the right and not the other way round.
They should acknowledge the Dunning-Kruger Effect and the damage it can do and focus on helping develop an agile mindset throughout the organisation, including its leadership and governance structures.
About the author
Simon Girvan FBCS CITP is an agile coach working in the Public Sector. He is a subject matter expert on Agile for the BCS and reviewed and contributed to the BCS book 'Agile and Business Analysis' by Lynda Girvan and Debra Paul.
He has spoken at various conferences including Agile Manchester and the Agile Business Conference. He blogs at Agile Centre. He is also on Twitter and LinkedIn.
