We all know the litany of software development project failures, with nearly 85 per cent of projects failing on some measure: projects delivered late, of low business value, poor quality or way over budget.
The software industry is great at producing the next generation in tools and technology, so why are we failing so badly?
Focusing on the right type of skills
The Higher Education Statistics Agency's 2003/04 data shows that about 72,000 students achieved their qualification in computer science or technology. What happens to these students' skills when they leave university?
Not much it seems. Most software developers are given the occasional language course or technology course but not much beyond that. For example who teaches developers about communicating with customers about requirements? This is largely learned on the job - if you are lucky. Some of the more fortunate get the opportunity to work with a strong team or find a personal mentor. But many - too many - simply fall through the cracks and never get the chance to really excel.
What's more we seem to be caught in a system that accepts failures, poor performance and frankly rubbish software. If the customer accepts poor-quality software that is over deadline - why should the software development team change? But this is a vicious cycle: failures lead to a lack of pride in work, leading to poor products and more failures.
We can do something about this by focusing on helping developers learn skills that really deliver high value to the business or customer. These sorts of best practices skills need investment and a process that can support them.
Taking a craftsmanship approach
In his 2002 book Software Craftsmanship Pete McBreen outlines a new paradigm for software engineering. The approach is to recognise that software is a craft: part art and part skill. Like the blacksmiths of their day, software developers should be measured on the quality of their work, their ability to deliver value to the business and be accountable for what they produce.
We have certainly seen this - the best software developers have a certain pride in their work, stand by what they have done and are willing to learn from others to improve. This environment of constant learning, best practice and high quality is the sort of environment we need to encourage in our software development organisations.
Let's also be clear. We are not saying every software developer must be a star. However, we have to give developers the tools and opportunity to meet their potential.
A process that supports the craftsman
One of the most flexible methods that allows for and supports the craftsman approach are agile methods. And while agile is not a cure-all and we need to be pragmatic in its application, it combines much of the best practices skills that developers should learn.
One of the criticisms of agile has been that it requires exceptionally skilled software developers. This reminds me of the website www.despair.com - a tongue-in-cheek site selling demotivational posters. One of the posters has a picture of the leaning Tower of Pisa with the caption: 'Mediocrity. It takes a lot less time and most people won't notice until it is too late.' Mediocre developers in any process are still mediocre. Agile has practices that allow people to achieve their potential, do good work or get out of the way.
Far from needing all stars, a mixed level of team members often works best. Agile software development uses pair programming and shared code ownership as practices to ensure quality. A wonderful side benefit from this is that those with experience can pair in with those with less experience, giving an opportunity for cross-learning and training. This also helps to increase knowledge across the team about the system, ensuring we don't develop dependencies on individuals, as dependency can put projects at serious risk.
Agile uses many team-based practices to increase visibility of the project, for example agile projects tend to have high levels of communication and feedback in-built due to practices such as daily stand-up meetings and frequent releases. This means that everyone on an agile team is effectively forced to up their game. That means even those with lacklustre performance or skills are forced to participate.
Having to up your game may sound like hard work - and it is; agile teams that we have seen are amongst the hardest working software teams out there but agile creates an environment of camaraderie and fun. One architect I know said, 'I feel re-invigorated in my job by working in an agile manner - it is actually fun and challenging again.'
Agile also encourages a lot of communication between the team, between the team and testers, between the team and customers, and so on. While communication may not be every developer's cup of tea, it is absolutely reasonable to expect that customers may want to speak to the people developing their software.
But we need to be pragmatic; we can't have dozens of customers making requests and we can't expect all developers to relish the opportunity to talk to customers. Teams need to be pragmatic; one team we worked with used a few of the development team for the communication roles with customers - though everyone was present at meetings.
And in the reverse we have seen customer teams use a single voice into the development team in order to explain requirements and changes. Do what works and try to communicate face to face when you can.
Importantly agile helps to bring skills gaps and communication issues to light - allowing the team and manager to address these issues. Too often teams hide from these issues, so they escalate. One thing is key here: in today's commercial software development world - whether internal IT or a software company - a lack of interpersonal skills will always be a problem regardless of the type of software development process, agile or otherwise.
Another aspect to consider is how software developers are recruited. Here is an ad I found recently:
'Software engineer, with understanding of engineering as well as programming required for software engineering company. Educated to degree level, three years experience of SW design and development. Preference is given to candidates with Java/C/C++ programming skills and experience of working on UNIX/LINUX platforms.'
There is nothing here about problem-solving, communication or the like. Hopefully that would be covered in the interview, but likely not. If we take on board that software is part art and part skill - then we should be asking for a developer's portfolio of work and ask them to demonstrate their skill. We encourage companies to ask candidates to submit a portfolio piece - a piece of work they are especially proud of. If they don't have one we give them a small project to work on and submit.
Successful candidates are brought in for an 'audition' of sorts - they actually pair in and code with the team. This gives the team a chance to see what the candidate can do and allows managers to see how a candidate will actually code. A personnel interview follows. We have on occasion even used psychometric tests to assess individuals. This all may seem intensive but when you consider there are multi-million pound projects on the line - the extra effort is often worth it.
So what can we achieve by changing our framework to a more agile / craftsmanship approach?
Learning - best practices and cross-skilling / skills transfer
A craftsmanship approach combined with agile methods allows teams to cross-skill in technologies and experience levels, bringing up the knowledge of the whole team. Further, more junior team members can learn quickly using a skills transfer approach from a craftsmanship model. Moreover as a development organisation we can start to look at how to capture this learning and create a set of development best practices that are mapped to our specific organisation and can be used across teams.
Morale - pride in work
People naturally want to do a good job and succeed. Agile methods can help foster a sense of teamwork and pride in work, boosting the overall morale. Team morale cannot be underestimated especially in today's climate of tight deadlines and heavy competition.
Business / IT relationship - deliver business value and communication
Craftsmen think of their customer first - always. The practices within the agile framework support this thinking and deliver high business value software, early and often to customers. The way agile does this is through lots of communication, ideally face to face. This collaborative effort in a project is very important to drive project success.
Management - visibility
Ask a developer where they are in a particular implementation, you'll get one of two answers: 'nearly done', or 'just started'. With agile methods managers get to see how developments are really doing, how close they are to completion and are able to react and manage any problems that might arise. Further, agile doesn't allow any room for hiding through daily stand-ups and information radiators; managers can assess any skills gaps and address these.
Quality - on time and on budget
Agile has repeatedly performed well on delivering quality and it is something that software craftsmen would not compromise on. One company we have worked with was able to get their software test cycle down to weeks rather then months - while increasing quality.
Quality over quantity - less can be more
In today's economy we have more to do with fewer resources. Giving software developers the right kind of skills means we can do this. I would take one developer who has the potential to be exceptional over two average developers any day.
What we need to do to fix our software processes is find Zen: assess our software development values, focus on quality and develop a craftsmanship approach to our work.
