A framework for Test Management
With the widespread adoption of capture/playback tools over the last five years, bolt-on products have emerged to help testers manage the volume of developed scripts. These test management products have, by design, provided a point solution for a specific testing tool and a specific type of test.
Point solutions versus frameworks
Many companies gaining experience in test automation technologies perceive a need to manage a wide range of tests (for example, manual, regression, unit, performance, and acceptance) and related testing tools. In markets such as telecommunications and networking, the testing tool utilised may even be a specialised piece of lab equipment such as a bulk call generator or a network/protocol analyser.
The importance of tool integration
To satisfy today's diverse range of requirements, any test management solution must accommodate different testing tools and different test types on demand. This suggests the notion of a generic test management framework into which any type of testing tool can be integrated and any type of test executed and controlled.
Integrating different tools into framework allows for communication of data in numerous productive ways. For example, the test execution engine of a test management framework can automatically populate many fields of a defect or problem report raised from a failed test.
Similarly, trace information can be gathered from running a suite of tests and instantly fed into a code coverage tool to give some indication of completeness. Integrating a configuration management tool into the same framework gives version control over tests and related data sets. When a test is a re-run, the correct versions of software and test data previously recorded can be automatically checked out prior to execution.
For mission-critical applications, a further integration with a requirements traceability tool would allow testers to correlate requirements with test cases and exceptions to be flagged to management (for example, an unfulfilled requirement due to no linked test cases).
A central data repository
The concept of a framework should be based on a central repository in which data such as test case information can be stored. Again, needs are diverse and can change even from one test team to another. Total configurability of both information entered and the Graphical User Interface is the order of the day.
Graphing, reporting and online documentation
Information stored in a repository is only useful if you can retrieve it readily in the required format. SQL reporting allows custom reports and graphs to be generated on demand to answer ad-hoc queries and serve as inputs to management reports.
Users require total control over content and format. Interactive reports and graphs generated from a central and up-to-date test repository are invaluable for indicating progress and bottlenecks in the testing process.
Conclusion
Test Management Frameworks are a natural evolution of test management technology in accommodating a wide range of tools and testing needs. A framework should cater for both in-house and commercial tools.
Tools integrated together into a coherent environment can benefit the user in a variety of ways. Since a central repository underpins the tools, management views on progress and testing bottlenecks can be readily established through reports and graphs.
Gary Wilson, European Technical Support Manager, Silicon Valley Networks a company specialising in Test Management products.
