It is not only IT that has changed dramatically over the past 50 years; business has changed too - and a lot of that change has been driven by us in IT.
Our technologies have opened the door to globalisation and accelerated changes in business process and practice, massively shortened product life cycles as globalisation has introduced new foreign entrants into our home markets and enabled us to outsource labour-intensive work into cheaper economies.
All this has changed the expectations and demands of the customers for our development products.
Modern business now typically develops new products in months instead of years, and commonly changes services and the business processes that support them in weeks instead of months.
The rate of change in business has undergone several step-changes over the past few decades, and in parallel business adoption of and dependence on our IT systems has deepened - IT is now commonly business-critical 24 x 365 and it has become difficult to make changes to business without implementing corresponding changes to the IT systems that support it.
In the face of this increased dependence on IT, the growth in business influence and responsibility held by IT functions and the weight of expectations placed upon them by their business counterparts, many IT leaders have increased their use of formal development management methods to help them assure reliability of delivery to the business - and they have fallen into two camps, Waterfall and Agile.
Waterfall has been with us a long time. The concept was described publicly over 50 years ago, and the Waterfall name attached to the sequential development model in the mid-70s. It is a proven model for the development of new software and systems, designed to ensure that the product of development is produced efficiently and matches the customer’s requirements.
Many of us in the IT profession rely on proper execution of the Waterfall method to help ensure that our projects deliver on our promises, captured in the initial stages of the project as requirements.
The requirements specification is not only the key reference tool for the development team, it also forms the primary contractual document between supplier and customer, freezing at a point in time the customer’s expression of their needs as a point of reference against which the eventual product may be measured.
As a method it can be argued that Waterfall has not served us in IT well, nor has it delighted our customers. Waterfall projects tend to seem lengthy, and it is common that during the development lifetime the customer’s requirements evolve, meaning that the end product fails to meet customer’s evolved needs. Despite the project successfully producing exactly what the customer asked for, the product disappoints and, consequently, our IT function disappoints.
Many IT functions and developers have reacted to this disappointment by abandoning the Waterfall method in favour of agile methods. ‘Agile’ has evolved over time and, whilst it was first given the name Agile in 2001, it had been around for many years prior. I personally have participated in projects run under what would now be called agile methods since the mid-80s. If you think about it that’s not very long after Waterfall was codified.
Agile is primarily characterised by its flexible approach to requirements. Requirements are captured at the beginning of a project, as in Waterfall, but typically in much less detail, and are clarified as the project progresses.
Agile projects typically progress as a series of short iterative developments instead of a long haul, with a pause for breath between each iteration during which the customer is shown the product so far and asked to prioritise and clarify requirements for the next iteration. Agile is essentially development bit by bit, assembling a jigsaw as the project progresses instead of the Waterfall approach of demanding a complete picture upfront.
Agile is probably no more of a recipe for project success than Waterfall, but its failings are different. Properly executed an Agile development will delight the customer because they will get what they want, even though their needs have changed during the life of the project. It may, in theory, have cost more and taken longer than originally envisaged - two of the key measures of project success - but customers generally forgive these failings if the product is right.
Most commonly successful Agile projects are shorter, less effort and cheaper than their Waterfall counterparts because the requirements specification activity is much less detailed and onerous, and also because the customer commonly says the product is good enough when it has fewer features than they originally envisaged or would have specified in a Waterfall development.
Set against these merits Agile is more difficult to manage, commonly requires more highly skilled and flexible developers, and has fewer, weaker role definitions, which makes the task of improving our developers’ skills more difficult.
Waterfall, a simple and reliable method, is still widely adopted despite its well exposed weaknesses, because it is achievable and manageable. Agile is a harder discipline for the IT leader to deploy.
The bigger the organisation the greater the contrasts between the methods become - implementing Waterfall on a large scale seems structured, disciplined and managed, whereas large-scale Agile can easily descend into anarchy. Consequently we have become a divided profession, with a predominant reliance on Waterfall in larger organisations and Agile being contained to smaller or more fragmented organisations. This is becoming a problem.
Given the contrasting natures of Waterfall and Agile and the increased dependence of business on IT, businesses that adopt Waterfall are commonly hampered and held back by the slow and stately progress of Waterfall development projects in comparison with competitors using Agile.
The Agile focus on producing ‘minimum viable product’ as soon as possible, and worrying about the bells and whistles later, enables its adopters to deliver business change and new products and services sooner and at lower cost.
When we look at the performance of businesses in the market the contrast has become starkly pronounced. Large businesses, giants of their industries that used to be leaders, have become followers, trying to hang on to previously established market share in the face of competition from an ever-increasing array of nimble upstart competitors that have seemingly appeared out of nowhere with superior offerings.
When we look below the surface and consider what these upstarts have done it is often clear that they have created new products and services in shorter timescales and with smaller budgets than the large players could ever imagine.
The Waterfall vs. Agile method debate is now being played out in the major industries and stock markets around the world. The division in our profession between Waterfall and Agile has become material; it is no longer a matter of management approach and philosophy, it has become a key factor in corporate survival, a choice between being quick or dead.
Large corporate IT leaders around the world over are realising the need to become Agile. CIOs and CTOs are succeeding or failing based upon their ability to introduce agility into their IT functions, and many are learning that it is far from easy.
Agile is not a single, simple coherent recipe, it is a philosophy. For an IT function to transition from Waterfall to Agile is not merely a matter of process change; it requires a complete change of culture and, as any decent business guru will tell you, culture is the single hardest thing to change in a workforce.
The Agile culture deprecates tightly defined roles, erodes the value and recognition of personal success, reduces certainty, stresses teamwork and continuous improvement, and demands constant close communication with customers - these are huge changes for what has become an IT profession characterised by tightly defined job roles.
But we don’t really have a choice. As IT leaders many, perhaps most, have recognised that, somehow, Agile is the way. Our business colleagues now demand IT responsiveness that can only be provided through the adoption of Agile, and they in turn have no choice because the markets they serve place similar demands upon them.
As we approach the second half of the decade our IT leadership agenda will be dominated by this simple ultimatum - Agility or redundancy, be quick or die.
The primary measure of an IT leader’s success will, between now and the end of the decade, not be how cheaply and reliably they can make the IT function deliver, but it will be how nimbly they can make it respond to the ever-changing needs of the business.
It is possible that the most influential attribute of the successful IT leader will not be anything to do with technology or information, but their ability to change and control the organisational and professional culture of the IT function.
