Adversarial training is a widely used technique to enhance the robustness of deep learning models. However, it is computationally expensive due to the need to calculate gradients for generating adversarial samples. While fast gradient sign method (FGSM)-based adversarial training is commonly employed to improve model robustness, it still incurs significant training time and computational overhead compared to standard training. Moreover, the robustness of models trained with FGSM-based adversarial training often remains unsatisfactory. Among the most efficient training techniques, mixed precision-based training reduces computational overheads by accelerating the standard training process. However, applying mixed precision to adversarial training does not reduce computational overheads and increase robustness. Thus, this article proposes a novel adaptive precision layering-based adversarial training and gradient accumulation method to reduce the computational cost. The proposed method efficiently assigns precision levels ( single and half precision ) to each model layer based on its sensitivity to precision loss. The less sensitive layers are assigned with half-precision, and the output layer is assigned with single precision. The perturbations are generated during the adversarial training by leveraging the fast gradient sign method with a random initialization technique. Additionally, the accumulation steps are also adaptively set based on the gradient flow and stability analysis. The adaptive precision layering accelerates the utilization of the graphics processing unit, and the efficient gradient accumulation method avoids unnecessary gradient calculation, thus reducing training time. By adaptively assigning the precision, gradient accumulation, and random initialization, we effectively solve three major problems in adversarial training: 1) significantly reduce the training time overheads, 2) solve the catastrophic overfitting, and 3) improve the robustness of the target models. Experimental evaluations on three state-of-the-art self-driving car models demonstrate an excellent improvement in adversarial robustness while significantly reducing the computational overheads.

Test automation intrusive to the devices under test is difficult to apply on closed or uncommon touch screen systems, e.g., a Switch game console or a digital instrument running a self-defined operating system. There is a lack of non-intrusive test automation techniques for situations where intrusive testing is impossible or not easy to apply. This paper presents RoScript, a novel robotic visual GUI testing system for truly non-intrusive test automation of touch screen applications. RoScript expresses GUI actions in visual test scripts and executes them via a physical robot. A key innovation of RoScript is a test engine armed with environment calibration techniques to achieve automated test execution without manually setting any environment parameter or adjusting the robot arms for a new subject under test. Additionally, two complementary computer vision-based methods are also introduced to record test scripts from videos of human actions on a touch screen. The RoScript test automation does not rely on the internal system of a device under test, making it truly non-intrusive and suitable for touch screen applications running on almost any platform. We evaluated RoScript on a diverse range of devices--including three Android/iOS phones, a Windows tablet, a Linux-based Raspberry Pi, a GoPro camera, and a Switch game console--across over 1100 GUI actions in 160 test scenarios. The results demonstrate RoScript’s high accuracy in test execution: 94% for executing test scripts and 97% for replicating GUI actions. Furthermore, RoScript accurately recorded about 85% of human touch screen actions into test code. These results highlight RoScript’s potential as a truly non-intrusive, cross-platform solution for GUI test automation.

Existing intrusive test automation techniques for touch screen applications (e.g., Appium and Sikuli) are difficult to work on many closed or uncommon systems, such as a GoPro. Being non-intrusive can largely extend the application scope of the test automation techniques. To this end, this paper presents RoScript, a truly non-intrusive test-script-driven robotic testing system for test automation of touch screen applications. RoScript leverages visual test scripts to express GUI actions on a touch screen application and uses a physical robot to drive automated test execution. To reduce the test script creation cost, a non-intrusive computer vision based technique is also introduced in RoScript to automatically record touch screen actions into test scripts from videos of human actions on the device under test. RoScript is applicable to touch screen applications running on almost arbitrary platforms, whatever the underlying operating systems or GUI frameworks are. We conducted experiments applying it to automate the testing of 21 touch screen applications on 6 different devices. The results show that RoScript is highly usable. In the experiments, it successfully automated 104 test scenarios containing over 650 different GUI actions on the subject applications. RoScript accurately performed GUI actions on over 90% of the test script executions and accurately recorded about 85% of human screen click actions into test code.