Test process management

Testing into the new millennium

Testing is increasing in profile and is no longer thought of as the 'dead end' task given over to people who can do nothing else. At long last, testing is recognised as a major aspect of system development and not just something to be done on an ad-hoc basis if time permits.

System failures are extremely costly. Benefits from education and tool support have come a long way to highlight both the importance of testing and where it can be integrated into the systems development life cycle.

Benefits have been seen from having independent test teams or test specialists and now more than ever emphasis is on 'Does the system meet the requirements?' It has now been recognised that the users of the system under test are required to be involved in the testing of the system.

Automated testing is becoming increasingly popular and has done a lot to increase the profile of testing. However, test automation benefits are only seen when used within a good test process. Without a good test process it is like software development without a development process.

Just as software development has matured from using unstructured processes and methods to structured practices, testing is now following the same path. A structured testing method will decrease many chances of error and will allow complex projects and systems to be managed more efficiently and effectively.

A Process Driven Approach to Testing - 'The Seven Steps'

1. Test Strategy - The test strategy defines the test stages and types of testing to be completed for the system under test. The report will contain agreed entry and exit criteria for every stage and there will also be an agreed key responsibility matrix.
2. Test Plan - This is typically a project plan including a Grant chart of when activities are scheduled to happen. It will also include project dependencies and resource requirements.
3. Analysis - 'WHAT to test'. The first step within the Analyse phase is to gather and review the reference data from which the testing will be derived. A functional decomposition can then be performed breaking down the system under test into functional areas.
   
   Each functional area can be assigned the appropriate business risk. Detailed measurable test conditions derived from agreed exit criteria are created for each functional area and are the focal point of the testing. Priorities can be assigned to these test conditions enabling more control. The reference data is linked to the functional areas providing traceability and allowing impact analysis.
4. Design - 'HOW to test'. Within the Design phase Test Specifications are created which could typically represent a business process. Within the Test Specifications, Test Scripts are created detailing the input data and expected results. Test Data can be recorded and linked to the test scripts. Every test scripts needs to clearly identify its purpose and reference the conditions which it is aiming to prove.
5. Schedule - 'WHEN to test scripts'. This phase enables the scheduling of test scripts appropriate for the particular software releases. The test schedule will reflect the appropriate structure necessary for management reporting.
6. Execute - 'Execute the scripts and record WHAT HAPPENED'. Scripts are executed and the results are recorded. Test condition Status records are created for every occurrence of a test condition within the scripts. This will allow the test coverage, success and failures to be reported on against the original requirements.
7. Manage the Process - 'INTRODUCE CONTROL of the whole process'. By introducing a structured approach to testing you are able to manage the whole process from initial requirements through to implementation. By keeping all the test criteria and associated risks, test scripts and other test assets related together you are able to sensibly manage the project providing the following information:
   
   - Identify % test coverage and success
   - Identify % of High Business Risk success
   - Manage the impact of change

Test Process Management encompasses all testing activities throughout the development life cycle. 'Real' test planning is part of this process, which can be tied into the Project Plan.

Test Analysis ties the planning and requirements into the creation and running of tests. Without it 'How do you know why or what you are testing?' The inclusion of test planning and analysis is quite clearly the way forward - Software testing in the millennium will use Test Process Management.

Management of change

By following the process driven approach to testing detailed above you will be able to manage change throughout the system development life cycle. By creating the relationships from requirements through to actual test results you are able to trace the effect if any requirements are changed.

Management reporting

The ability to provide information at all stages within the test process is crucial for controlling project progress. Statistics need to be derived and related from the later stages in the process all the way up to the Strategy and Plan. This has been notoriously difficult to achieve in the history of software development resulting in countless high profile project failures.

When to stop Testing - Exit criteria

Exit criteria is the focal point of testing. Without exit criteria how can you know 'When all testing is complete?' Exit criteria provides the basis for useful and sensible measures of coverage and success.

Without exit criteria all coverage and success measures are meaningless. For example coverage or success, measured in terms of scripts passed is meaningless if you are unable to relate it back to the original requirements.

However the requirements are usually written at a very high level, especially for User Acceptance Testing and System Testing. It is often necessary to break down each requirement creating more detailed test criteria.

This may not apply as often to lower level testing such as unit testing and link testing where the requirements take the form of technical specifications and program specifications. These will normally be specified at a lower, more detailed level in the first place.

Creating test scripts from requirements does not allow you to maintain any relationships between the requirements and the more detailed test criteria.

Exit criteria - Meet the need

The Analysis phase allows you to maintain the relationships required.

Requirement test condition test script

The diagram above shows that a requirement can be broken down into many test conditions and that each test condition may be tested in more than one test script.

When you are reporting on the testing status showing the coverage, success and failures a more detailed and accurate picture will be created.

Tools to support test process management

Many testing tools on the market address the automation of testing. These tools usually have a component termed 'Test Management' or 'Test Planning'.

These tools support Test Execution Management very effectively but give little support to the higher level activities identified in Test Process Management.

Most organisations often struggle to control the test process with a mixture of Spreadsheets, Word Processors and 'home-grown' Databases. However, there are a few proprietary tools which can more effectively answer questions such as:

• Is the system ready to go live?
• If we go live now what risk is associated with that?
• What coverage have we achieved in our testing to date?
• What success have we achieved to date?
• How much more is there to do?
• Can I prove the system is really tested?
• What is the impact of this change to the system and what must be re-tested?

If you would like more information about T-Plan Professional or a ‘White Paper’ on Test Process Management, please contact:

Mike Smith, Test Management Solutions Ltd