How mature is your automated test suite?

You may be familiar with maturity levels used by the SEI's Capability Maturity Model (CMM). For maturity of an automated test suite, our levels are on a human scale: infant; adolescent; and adult.

An automated test suite at the infant stage will need a lot of baby-sitting. Many tests will not run to completion, partly due to bugs in the software under test (and of course there will be a good number of them) and partly due to the test script being out of step with it.

Bugs in the software will not only cause the test to fail but will also probably cause the script to stop executing. Inconsistencies between the software under test and the test script will also invalidate anything else the script tries to do.

In any case you have to fix the bug or update the script and then re-run the test to find the next problem. You will probably spend more time working on the test scripts than thinking about testing.

This is baby-sitting at its best (or worst!) - you cannot leave the tests running unattended for more than a few minutes at a time. (Doesn't sound much like 'automation' does it?).

If you're lucky you won't spend much more time encouraging the script to run to completion than you would have done running the test manually in the first place. Unfortunately though, such luck is rare! Automating a test for the first time typically takes two to ten times longer than running that same test manually.

At the adolescent stage the test suite can be left alone to execute the tests unattended for a reasonable time, perhaps a few hours, even over-night. But a single error in the software will cause a lot of tests to fail.

The error may be important to know about but we don't need dozens of tests telling us about the same one. A lot of time can be wasted analysing the cause of each failure and these failures prevent many tests revealing other faults that we don't know about.

To find these faults it is necessary to fix the problems and re-run the tests, a cycle that may have to be repeated many times. Just as with adolescents, sometimes the test suite is surprisingly useful; at other times it behaves irresponsibly and can cause of lot of damage.

A mature test suite is trust-worthy. It can be left to run unattended for a long time, even over a whole weekend. At the end of the test run there will be a lot of useful information. Some tests will have failed (perhaps most of them) but, by and large, they will all tell you something different and mostly about the software under test, not about the test script.

Now you can concentrate on finding and fixing problems in the software. Your test suite has done its job without your intervention. Furthermore, most, if not all, of the environment set up and clear down before and after testing will be performed by the test suite. You will be free to concentrate on other more creative tasks.

Each test suite will have to grow up through these stages but you don't have to let nature take its course. With a bit of thought and careful nurturing (and a good book*) you can speed it up.

Implementing an efficient and effective automated test suite is not an art but an engineering discipline in its own right. The fallacy of creating automated tests using record and playback techniques alone is now fairly well understood, but that is only the first lesson.

There is more to a mature test suite than automated test validation, data-driven scripts and test completion reports. It involves a planned approach, test design and test maintenance. Only by managing all aspects of test automation will the full benefits be realised.

Make sure your automated test suite grows up and gives you all the benefits you deserve.

* Software Test Automation, Mark Fewster and Dorothy Graham, Addison Wesley, 1999, ISBN 0-201-33140-3

Mark Fewster, Grove Consultants, 1999