Scriptless testing generates test sequences dynamically by automatically exploring the Graphical User Interface (GUI). Instead of relying on predefined scripts-which have proven expensive to maintain-scriptless tools detect available widgets, derive possible actions, and select actions on the fly using exploratory techniques such as random selection, model-based inference, or reinforcement learning. While scriptless testing is a valuable complement to scripted approaches, current techniques lack the intelligence needed to strategically select and execute GUI actions that fulfill specific functional testing goals-such as those derived from requirements, use cases, or user stories. Unsurprisingly, this leads companies to question the viability of scriptless testing tools and continue relying on scripts for test automation. This paper reports on the integration of Large Language Models (LLMs) into a scriptless GUI testing tool for action selection, aiming to determine whether it can generate effective action sequences to test specific functional requirements. Our results demonstrate that a multi-objective test goal structure, combined with historical and feedback context, enables LLM-empowered scriptless testing to automate functional testing. Although further research is needed to tackle challenges in complex test scenarios, our findings offer promising results that LLM-empowered scriptless testing can reduce reliance on the maintenance-heavy aspects of traditional scripted testing.

Computer 3D games are complex software environments that require novel testing processes to ensure high-quality standards. The Intelligent Verification/Validation for Extended Reality Based Systems ( iv4XR ) framework addresses this need by enabling the implementation of autonomous agents to automate game testing scenarios. This framework facilitates the automation of regression test cases for complex 3D games like Space Engineers. Nevertheless, the technical expertise required to define test scripts using iv4XR can constrain seamless collaboration between developers and testers. This paper reports how integrating a Behavior-Driven Development (BDD) approach with the iv4XR framework allows the industrial company behind Space Engineers to automate regression testing. The success of this industrial collaboration has inspired the iv4XR team to integrate the BDD approach to improve the automation of play-testing for the experimental 3D game LabRecruits. Furthermore, the iv4XR framework has been extended with tactical programming to enable the automation of long-play test scenarios in Space Engineers. These results underscore the versatility of the iv4XR framework in supporting diverse testing approaches while showcasing how BDD empowers users to create, manage, and execute automated game tests using comprehensive and human-readable statements.

Scriptless testing at the GUI level involves generating test sequences on the fly. These test sequences mimic user interactions on the GUI. The creation of these sequences works through action selection rules, which is most commonly based on stochastic methods. Script-less tests are reliable because they work with the actual state of the System Under Test (SUT). However, the tests are less specific, harder to interpret, and it is difficult to test concrete use cases or workflows. We want to tackle this drawback of scriptless tests by introducing action selection rules that are easier to guide than pure stochastic methods. In this paper, a new approach based on a grammar for the action selection rules is proposed, enabling scriptless testing tools to mimic user behaviour when interacting with web applications. While grammars have been used in software testing to generate input data for test cases, the proposed approach uses grammars to specify action selection rules to generate test sequences that mimic testing strategies employed by human testers. An empirical study has been performed to evaluate the effectiveness and the efficiency of the grammar-based action selection rules to filing web forms in comparison with random action selection rules. In the study, two SUTs were used: WebformSUT and Parabank. The average success rate for the grammar-based approach was 95.9% against random's 57.0% for WebformSUT and 99.8% against 55.7% for Parabank. For the widget interaction grammar-based had an average deviation from the ideal ratio of 0.06165 (WebformSUT) and 0.0180 (Parabank), compared random's 0.4318 (WebformSUT) and 0.7774 (Parabank). The results demonstrate the effectiveness of the grammar-based approach and the improvement in the use of resources.

Software testing is an important part of the software development lifecycle. As it is a highly sought-after skill in the industry, it is not surprising that there has been a great deal of research into the teaching of software testing in higher education. Most of this research proposes or evaluates pedagogical approaches or software testing tools to assist teachers in educating the next generation of software engineers. These evaluations are often limited to measuring teachers’ opinions about the use of a novel pedagogical approach or an educational tool and students’ acceptance and performance in terms of desired software testing skills. While tools and pedagogical approaches address specific aspects of a course, to date, little attention has been paid to the opinions of the students about all the individual aspects of a software testing course. This paper aims to address this missing student perspective by taking a holistic view of software testing course designs. To address this gap, an exploratory study was performed by distributing a questionnaire to 103 students from ten different courses to gauge their opinions on a software testing course they are enrolled in. The results show that students generally have a positive perception of the different aspects of their software testing course. However, several areas for improvement were suggested based on the gathered data.

Computer games have reached unprecedented importance, exceeding two billion users in the early 2020s. Human game testers bring invaluable expertise to evaluate complex games like 3D sandbox games. However, the sheer scale and diversity of game content constrain their ability to explore all scenarios manually. Recognizing the significance and inherent complexity of game testing, our research aims to investigate new automated testing approaches. To achieve this goal, we have integrated scriptless testing into the industrial game Space Engineers, enabling an automated approach to explore and test sandbox game scenarios. Our approach involves the development of a Space Engineers-plugin, leveraging the Intelligent Verification and Validation for Extended Reality-Based Systems (IV4XR) framework and extending the capabilities of the open-source scriptless testing tool TESTAR. Through this research, we unveil the potential of a scriptless agent to explore 3D sandbox game scenarios autonomously. Results demonstrate the effectiveness of an autonomous scriptless agent in achieving spatial coverage when exploring and (dis)covering elements within the 3D sandbox game.

State model inference of software applications through the Graphical User Interface (GUI) is a technique that identifies GUI states and transitions, and maps them into a model. Scriptless GUI testing tools can benefit substantially from the availability of these state models, for example, to improve the exploration, or have sophisticated test oracles. However, inferring models for large systems requires a long execution time. Our goal is to improve the speed of the state model inference process. To achieve this goal, this paper presents a distributed state model inference approach with an open source scriptless GUI testing tool. Moreover, in order to be able to infer a suitable model, we design a set of strategies to deal with abstraction challenges and to distinguish GUI states and transitions in the model. To validate it, we conduct an experiment with two open source web applications that have been tested with the distributed architecture using one to six Docker containers sharing the same state model. With the obtained results, we can conclude that it is feasible to infer a model with a distributed approach and that using the distributed approach reduces the time required for inferring a state model.

Traditionally, end-to-end testing of mobile apps is either performed manually or automated with test scripts. However, manual GUI testing is expensive and slow, and test scripts are fragile for GUI changes, resulting in high maintenance costs. Scriptless testing attempts to address the costs associated with GUI testing. Existing scriptless approaches for mobile testing do not seem to fit the requirements of the industry, specifically those of the ING. This study presents an extension to open source TESTAR tool to support scriptless GUI testing of Android and iOS applications. We present an initial validation of the tool on an industrial setting at the ING. From the validation, we determine that the extended TESTAR outperforms two other state-of-the-art scriptless testing tools for Android in terms of code coverage, and achieves similar performance as the scripted test automation already in use at the ING. Moreover, we see that the scriptless approach covers parts of the application under test that the existing test scripts did not cover, showing the complementarity of the approaches, providing more value for the testers.

Summary Covering all the possible paths of the graphical user interface (GUI) with test scripts would take too much effort and result in serious maintenance issues. We propose complementing scripted testing with scriptless test automation using the open‐source testar tool. This paper gives a comprehensive overview of testar and its latest extensions together with the ongoing and future research. With this paper, we hope we can help and encourage other researchers to use testar for their GUI testing‐related research and pave the way for an international research agenda in GUI testing built upon stable and open‐source infrastructure.