Test automation intrusive to the devices under test is difficult to apply on closed or uncommon touch screen systems, e.g., a Switch game console or a digital instrument running a self-defined operating system. There is a lack of non-intrusive test automation techniques for situations where intrusive testing is impossible or not easy to apply. This paper presents RoScript, a novel robotic visual GUI testing system for truly non-intrusive test automation of touch screen applications. RoScript expresses GUI actions in visual test scripts and executes them via a physical robot. A key innovation of RoScript is a test engine armed with environment calibration techniques to achieve automated test execution without manually setting any environment parameter or adjusting the robot arms for a new subject under test. Additionally, two complementary computer vision-based methods are also introduced to record test scripts from videos of human actions on a touch screen. The RoScript test automation does not rely on the internal system of a device under test, making it truly non-intrusive and suitable for touch screen applications running on almost any platform. We evaluated RoScript on a diverse range of devices--including three Android/iOS phones, a Windows tablet, a Linux-based Raspberry Pi, a GoPro camera, and a Switch game console--across over 1100 GUI actions in 160 test scenarios. The results demonstrate RoScript’s high accuracy in test execution: 94% for executing test scripts and 97% for replicating GUI actions. Furthermore, RoScript accurately recorded about 85% of human touch screen actions into test code. These results highlight RoScript’s potential as a truly non-intrusive, cross-platform solution for GUI test automation.

Existing intrusive test automation techniques for touch screen applications (e.g., Appium and Sikuli) are difficult to work on many closed or uncommon systems, such as a GoPro. Being non-intrusive can largely extend the application scope of the test automation techniques. To this end, this paper presents RoScript, a truly non-intrusive test-script-driven robotic testing system for test automation of touch screen applications. RoScript leverages visual test scripts to express GUI actions on a touch screen application and uses a physical robot to drive automated test execution. To reduce the test script creation cost, a non-intrusive computer vision based technique is also introduced in RoScript to automatically record touch screen actions into test scripts from videos of human actions on the device under test. RoScript is applicable to touch screen applications running on almost arbitrary platforms, whatever the underlying operating systems or GUI frameworks are. We conducted experiments applying it to automate the testing of 21 touch screen applications on 6 different devices. The results show that RoScript is highly usable. In the experiments, it successfully automated 104 test scenarios containing over 650 different GUI actions on the subject applications. RoScript accurately performed GUI actions on over 90% of the test script executions and accurately recorded about 85% of human screen click actions into test code.