DevOps, the coalescence of agile systems administration and operations with development resources and staff, throughout all stages of the software production life cycle, is a reflection of how important operations has become in an increasingly ‘everything-as-a-service’ IT world. DevOps, which relies on a team-based approach to delivering software and services, looks like a significant departure from earlier methods, and integrating DevOps into your organisation’s IT delivery may prove critical to ensuring continued project management success.
As a computer software expert witness I spent much of a whole year investigating why a major IT outsourcing deal broke down and gave rise to the largest software contract dispute to come to the English High Court. At the heart of the dispute was a failed software development and implementation project, within the context of a major outsourcing agreement.
That agreement was effectively for DevOps service delivery, with the software development project being the key to managing and performing that delivery. It entailed the building of one new, single complex system, priced in the tens of millions of pounds, intended to replace many legacy systems. The outsourcing arrangement was to stretch over ten years; when it broke down, it gave rise to a legal action for recovery in excess of £200m.
The irony is that the agile systems administration ‘Ops’ part of the outsourcing deal was going well. But everything got bogged down in the software development phase; and, because the software implementation project was tied into the main contract, when the software development ‘went DisPro’, the entire DevOps outsourcing deal was canned.
There are clear lessons to be learnt from my expert investigations on this and many other IT disputes and litigation over DISPROs during the past 25 years. Here are a few brief bullet points of advice, towards better legal contracting for professional quality assurance of, and corporate investment in, major DevOps projects.
Addressing and managing the risks - Contract risk
- Split DevOps outsourcing contracts from development contracts. Outsourcing contracts often have software development and ‘value-add’ features built into them, with the lawyers who draft them believing they can put controls and checks in place. However, the software development often becomes problematic simply because software engineering remains one of the most complex and challenging activities undertaken. All the Ops service-level and other targets of the outsourcing may be readily achievable, but such systems admin and maintenance delivery requires completely different skillsets to software design and development.
- Have a significant retention in the DevOps development contract against final acceptance of the total developed software. Some payment should be retained until the software is well up and running. For a large contract it is not unreasonable that a percentage of the fee be kept back for a full year in case anything comes up in the software service delivery.
- Make the acceptance criteria objective, comprehensive and relevant to the true/most important needs of the business. How can you assess what is the quality of what is provided, if you were not sure what you wanted in the first place? People still seem to have problems defining and analysing system requirements and are bad at articulating what they want - with DevOps methodology many are actually likely to get worse at it.
- Try and go for a two-stage, conditional DevOps development programme, with the acceptance of adequate delivery of a ‘proving’ or ‘prototype’ system being required before activating the contract for a full system. This also provides an early test of the supplier. Design prototypes, ‘proofs of concepts’, construction maquettes of a system, together with ‘running walkthroughs’ of the Ops delivery profile, can all be produced within weeks and can reveal clearly the problems that may lie ahead on scale-up, in a small and contained way.
- When replacing / updating missioncritical legacy systems, never go live without a sensible trial period of parallel running. This should speak for itself, but one case in which I was involved concerned a failed integrated accounting software system DevOps implementation where a major consultancy had advised ‘go-live’ without any period of parallel-running - and also recommended that this ‘leap in the dark’ cutover should coincide with the end of the company’s financial year.
Technology risk
- Innovation is good, but it should not be done at a cost and risk to the customer. It has been known for a supplier to promise immediate use of new technologies and skillsets, and then, when it is awarded the contract, to spend the first months frantically trying to recruit the right people and bring them painfully up to speed. If the supplier is going to learn your business, new technology or tools and how to manage complex DevOps projects at your expense, this may be worth doing on a shared financingrisk-return basis. However, recognise that you are then pioneering and effectively being a venture capitalist.
- Wherever possible, use experienced DevOps professionals, with good references. Check-out supplier and staff CVs, and background credentials. The usual system ‘spec & select’ process often arrives only as a ‘least worse’ choice.
Financial risk
- Have a fall-back plan, review it and check it frequently. If you are outsourcing DevOps you need to ask: ‘Can I transition back in-house and return to my legacy systems if all else fails?’ The fall-back may be more expensive than the original development plan, but it could save your business if all goes hopelessly wrong.
- Just because you have outsourced your DevOps systems project doesn’t mean you have got rid of the risk. Not at all - you have just transferred it!
Learning points - Phasing
- Split every sizeable DevOps project into phases. Ensure tangible deliverables, of manageable and usable software services, within reasonable time-periods.
- The first phase should ideally be the ‘proving’ system delivery. Be resolute: if that simple first phase cannot be completed wholly satisfactorily, bite the abandonment bullet - things could only get worse later if you do not.
- Understand the contractual, technical and methodological distinctions between a ‘phase’ and a ‘release’ of software. A DevOps supplier may only make small changes to the software and call it a ‘release’. Do not assume that you are getting ‘released’ something drastically improved, with major progress being made in phases of the development. Furthermore, developers and operations staff see release management in rather differing ways: developers see releases as code, functionality, fault-repair and capability. Operations, however, see releases as a fluid movement of code services to recipients and users. There are clearly multiple opportunities for poor communications and vagueness in accountability.
Independent monitoring
- Have an independent ‘monitor’ as an integral part of any significant software DevOps contract / project. Significant construction engineering contracts and those in other professional disciplines routinely have the role of e.g. ‘The engineer’ built into them: someone who, ‘as an expert, not an arbitrator’, provides binding adjudication on, or other resolution of, all kinds of technical and project management matters and issues, quickly, as the project develops.
- Such a ‘monitor’, should be a senior, experienced, authoritative, mature IT consultant / project manager. Equally, it is vital that a company which has outsourced all or much of its IT DevOps keeps someone senior, skilled and experienced in-house / contracted to oversee whether the outsourcing / software development is being done well or not.
Board-level ownership
- It is vital that the customer has a DevOps project champion at board level. Modern digital systems- dependent businesses need boardlevel budget and responsibility for mission-critical DevOps systems developments.
- Such a board member needs to be someone who can peer-relate and negotiate with the DevOps supplier’s board directors, and also someone who won’t be bamboozled by suppliers, or in-house IT management or staff, ‘techies’. He / she needs to be an expert not only in the customer’s business, but in the tried and proven principles of good IT and software engineering/systems development and management methodologies - a true IT professional.
Some quick do’s and don’ts - ‘eternal truths’
- DO ensure a complete and correct requirements specification from the start. ‘How can you get what you want, if you don’t know what you want?’
- DO divide the work into small incremental deliverable modules. Ensure at least ten modules, approximately evenly spaced, so that the DevOps project cannot depart from plan by more than 10 per cent without the departure being obvious.
- DO arrange that each delivered DevOps service module depends only upon its predecessors; the system is then operational, with restricted functionality, and making a return on investment, from an early stage in the development.
- DO use every available means to ‘illustrate’ what has been specified (especially prototyping, animation, modelling, story-boarding, ‘running walkthroughs’).
- DO invest in the right skills.
- DO reduce risk and change as much as possible. Never introduce several new technologies together in the same project. DevOps software development is risky enough in its own right without building further risks, such as using a totally new, untried language or toolset, into the ongoing work.
- DON’T assume the first design is correct – be prepared for change!
- DON’T leave non-functional testing to the end.
- DON’T over-engineer - simple solutions are more elegant, have fewer potential weak-spots, and are easier to understand, change and maintain.
DON’T have large development teams - small teams work well: recall the ‘magic number 7’.
Unwanted questions
The DevOps project manager needs to develop an attitude and awareness which constantly asks the (unpalatable) questions: ‘If this DevOps software design/ construction / implementation job / project were terminated today (whether I wish for that to happen or not), and a court were asked to make a judgment as to the quality of the software development and project management procedures and deliverables which are in progress (in accordance with the contract with which all parties are supposed to be complying, and whether the job / project / work is currently completed or not) what would that court’s decision be likely to be?’
and
‘What technical or other evidence could and should there be readily available to assist the court in making its determination; and how should easily-understood analyses of and arguments based on that evidence be presented which would inevitably lead to a compelling (because objectively justifiable) conclusion in favour of the decision I genuinely believe is the correct one the court ought to reach?’.
What would the draft answers be, right now, for your DevOps project(s)...?
