Ghost in the machine: Automated Testing

The myth of record/playback

As a consequence, the way these tools are used has begun to change and more effective automation architectures are beginning to be developed. In this article, the first in an occasional series, John Kent looks at the tools and examines one approach-Record/Playback.

Automated testers are in a fortunate position in that they can have their tests running overnight while they are at home having a beer. Automated testers do it with their feet up-they can work when they are not at work.

That is a productivity level which no one else can match and a great reason for being in this exciting area of testing. This is the promise of automation: greatly increased productivity during test execution. Software is becoming so complex that the cost of adequately testing every release of a product is becoming prohibitive for many development organisations.

Good software is extremely expensive to create and testing it 'fully' could cost just as much again. The industry needs ways to do the repetitive, boring test execution tasks. It needs a ghost in the machine that will replay your tests.

Where are we now?

Has the promise of test automation been fulfilled? Unfortunately, not for most. The promised increase in testing productivity has been attained only by very few of those who have taken up the challenge of automated testing. This has been for a variety of reasons. Test automation is a very young branch of computing and it is easy to get it wrong.

Many people feel that this lack of success has been due to use of the wrong tools and there has been a lot of noise about tool evaluation in the testing community. But surely we can't all be using the wrong tools, can we?

Many of the problems with automated testing are primarily due to the approach adopted rather than the fault of any particular tool. The test automation community is coming to realise that the structure or architecture of the automated test programs and data has a major influence on the success or otherwise of an automation project.

The tools

Tools for automated testing of user interfaces were first built for text-based, mainframe systems were called 'Capture/Playback' (or 'Record/Playback') because recordings of each screen were captured when the user pressed the Enter key (or a function, 'F' key). The tests were recordings of what was sent to the processor on pressing the Enter Key and the response that was sent back by the software under test (SUT).

Although some of these tools were quite basic, they could be made to work, and work well. The author was involved in one test project on a mainframe system where recordings of over 100,000 screens were successfully used to find bags during regression testing.

With the introduction of PCs and then GUIs, the automated testing tools had to take a new approach. This was because with PCs the user/software interaction is much closer than with a mainframe.

For example, in a GUI the application software is invoked not just when the user presses Enter - a GUI system can respond to any key press or mouse click or even just putting the mouse over a particular part of a window.

GUI software is said to be event-driven because the application responds to a variety of user actions - i.e. it is driven by user created events. Because of this closer user/software interaction, automated test tools could no longer simply play-back the text on a screen; rather, they needed to be able to play-back the users’ actions.

The only way to do this is to have a list of actions which are performed on the SUT (System Under Test). These lists were called Scripts and thus the tools became script-based. Scripts contain actions such as Type 'Smith' in the surname text box and Click the OK button. In Visual Test, for example, a script might look like this:

Play "Smith"
WButtonClick("&OK")

These scripts are actually written in programming languages and you can write scripts manually as well-this is often called Scripting but perhaps ought to be called programming.

Automated test methods

Because these tools come with their own programming languages, they can be used in a variety of ways and that is what this series is really about-how do you use the tools for the best results?

Paradigm lost: record playback

Automated testing tools are often called 'capture/Playback' or 'Record/Playback'. It sounds great doesn't it? "All you have to do is simply record your tests and replay them whenever you like."

This, unfortunately is not the whole truth. Record and playback looks great, but it only really does what it is supposed to do in one situation: the tool vendor's sales demo. There is a growing body of opinion in the test automation community that recognises that it is not practical to create automated tests by simply recording them with script-based tools. I'll try to demonstrate why.

Recording produces unmaintainable scripts that consist of a long list of references to objects on your user interface. As an example, I recorded typing the previous sentence with an automated test tool (Visual Test) and this is what the recording looks like;

CurrentWindow = WFndWndC("Microsoft Word – QW99Pap.rtf)
Play "{Click 538, 0, Left}"
Sleep(2.374)
Play "{ENTER}Recording pr"
Play "oduces I{BS} unmaintaai{BS}{BS}inable scripts"
Play " that"
Play " "
Play "consist of a long list of references to"
Play "objects on your ue{BS}ser interface"
Play "."

You can see the test data-the sentence-is divided amongst several 'Play' statements. The '{BS}'s are where I made a keying mistake and then typed a backspace. In order to illustrate Record/Playback with a 'live' demonstration, I'll try to play this back (it should type the next sentence) and see what I get - here we go:

Recording produces unmaintainable scripts that consist of a long list of references to objects on your user interface.

Brilliant, so it works. So now I’ll underline the word unmaintainable here and italicise by selecting it with the mouse and clicking the Underline and Italics buttons. This is what I get when recording with Visual Test:

Play "{BtnDown 409, 136, Left}"
Sleep(5.232)
Play "{BtnUp 513, 137, Left}"
Sleep(2.106)
Play "{Click 302, 53, Left}"
Play "{Click 320, 53, Left}"

I tried to relay this and eventually, after some editing the script got it to work. I had to remove the first sleep statement in order ensure that the word was selected and of course before I ran it I took the underline and italics off in order to return to the initial state. It worked after editing, but look at the script.

Not only is it unpleasant code, it is also unmaintainable (well, at least very difficult to maintain) because it contains XY co-ordinates which makes it almost impossible to identify what it is doing and to which objects in the window. Also, we now have data hard-coded into our test program which we know from software engineering is not good a good approach to building software.

This example also illustrates one of the major arguments against Record/Playback which is that re-runnable tests must cope with what may happen, not simply what has happened. For example, one of the objectives of running tests against a system is to find errors that were not there before.

The automated tests must have code that recognises errors and this must be programmed into them. With a recorded script, if an error is encountered during a test run it is likely that the whole test schedule will come to a stop, or, as in our example, the test will continue blindly-completely unaware of anything wrong.

It is indeed ironic that, although the objective of automated testing is to find bugs, if the SUT behaves in an unexpected way -– i.e. there are bugs - then these recorded tests will not work. A test cannot simply be a recording played back into the SUT; rather it is an interaction between the test tool and the SUT.

You can use recording as a way of helping to create automated tests but you will have to take them and modify them by programming. Record/Playback tools are not really Record/Playback at all, they are Program/Playback tools. Try recording just ten minutes of input with one of the modern GUI script-based tools and see how well it replays.

Without knowing anything about your system I can guarantee that it will not run all the way through. Another enlightening exercise is to try to fix the recorded script in order to see what you would have to do to get it to replay correctly and the effort involved in doing this.

Once you accept that Record/Playback is primarily a sales gimmick for script-based automation tools, you have to accept that you will need to do some programming in order to create automated tests.

John Kent, CISS Ltd

References

Bach, James: Test Automation Snake Oil – http://www.stlabs.com
Kent, John: Advanced Automated Testing Architectures – Quality Week 1999
Kaner, Cem: Improving the Maintainability of Automated Test Suites – Quality Week 1997

© CISS Ltd