Recent advances in web agent technologies have enabled automated web browser operations through natural language instructions. While these technologies show promise for end-to-end test automation, significant challenges remain, such as uncertainty in test execution due to LLMs, increased execution time and cost, and decreased accuracy in identifying operation targets as web pages become more complex. To address these challenges, we propose XTestGen, which generates Gherkinformat test cases and JavaScript step definitions from natural language. XTestGen improves reproducibility by producing deterministic scripts, enhances maintainability through modular step reuse and scenario abstraction, and increases element identification accuracy in complex web pages using hierarchical tree exploration. Our evaluation shows that XTestGen enables abstraction and reuse in test generation and achieves higher accuracy in element identification than naive approaches. A demonstration video is available at: https://youtu.be/sQmsNCPGtPo

Black-box testing is vital for software quality assurance, focusing on system behavior without internal details. Cause-Effect Graphs (CEGs) visualize relationships between inputs and outputs, aiding in complex logical scenario testing. Despite their usefulness, creating CEGs is labor-intensive and requires expertise. This paper introduces CEGen, a novel method using Large Language Models (LLMs) to generate CEGs from natural language. CEGen employs LLMs to create truth tables from specifications and constructs CEGs algorithmically, making the process accessible to non-experts. Evaluation on ten problems from a tester training book shows that CEGen successfully generated error-free CEGs in 52% of attempts.

Exploratory testing is an important activity for improving software quality. Despite this, few studies have been conducted to support exploratory testing. One reason for this is the lack of available data on exploratory testing activities. We propose LatteArt, a tool to support developers in recording, visualizing and managing exploratory testing activities. LatteArt has a rich user interface and a variety of features to support exploratory testing, and is publicly available. We conducted a questionnaire with three developers to assess whether LatteArt addresses the challenges of exploratory testing. In our questionnaire, two or more developers indicated that LatteArt helps to address 14 of the 17 exploratory testing weaknesses identified in the existing literature. The test activity data recorded by LatteArt is easily accessible via REST APIs, allowing researchers to use the data and facilitate exploratory testing research. Our tool and demo are available at: https://github.com/latteart-org/latteart

End-to-end test automation is critical in modern web application development. However, test automation techniques used in industry face challenges in implementing and maintaining test scripts. It is difficult to determine and maintain the locators needed by test scripts to identify web elements on web pages. The reason is that locators depend on the metadata of web elements and the structure of each web page. One effective way to solve such a problem of locators is to allow test cases written in natural language to be executed without test scripts. In this study, we propose a technique to identify web elements that should be operated on a web page by interpreting natural-language-like test cases. The test cases are written in a domain-specific language that independents on the metadata of web elements and the structural information of web pages. We leverage natural language processing techniques to understand the semantics of web elements. We also create heuristic search algorithms to explore web pages and find promising test procedures. To evaluate the proposed technique, we applied it to test cases for two open-source web applications. The experimental results show that our technique was able to successfully identify about 94% of web elements to be operated in the test cases. Our approach also succeeded in identifying all the web elements that were operated in 68% of the test cases.

Automated program generation (APG) is a concept of automatically making a computer program. Toward this goal, transferring automated program repair (APR) to APG can be considered. APR modifies the buggy input source code to pass all test cases. APG regards empty source code as initially failing all test cases, i.e., containing multiple bugs. Search-based APR repeatedly generates program variants and evaluates them. Many traditional APR systems evaluate the fitness of variants based on the number of passing test cases. However, when source code contains multiple bugs, this fitness function lacks the expressive power of variants. In this paper, we propose the application of a multi-objective genetic algorithm to APR in order to improve efficiency. We also propose a new crossover method that combines two variants with complementary test results, taking advantage of the high expressive power of multi-objective genetic algorithms for evaluation. We tested the effectiveness of the proposed method on competitive programming tasks. The obtained results showed significant differences in the number of successful trials and the required generation time.

End-to-end test automation is important in modern web application development. However, existing test automation techniques have challenges in implementing and maintaining test scripts. It is difficult to keep correct locators, which test scripts require to identify web elements on web pages. The reason is that locators depend on the metadata in web elements or the structure of each web page. One efficient way to solve the problem of locators is to make test cases written in natural language executable without test scripts. As the first step of script-free testing, we propose a technique to identify web elements to be operated and to determine test procedures by interpreting test cases. The test cases are written in a domain-specific language without relying on the metadata of web elements or the structural information of web pages. We leverage natural language processing techniques to understand the semantics of web elements. We also create heuristic search algorithms to find promising test procedures. To evaluate our proposed technique, we applied it to two open-source web applications. The experimental results show that our technique successfully identified 94% of web elements to be operated in the test cases.

Scripted testing and exploratory testing are widely adopted among developers. It is known that exploratory testing can detect bugs more efficiently than scripted testing. However, exploratory testing has the problem that it is difficult to agree with third parties on test results and test quality. In this study, we propose a novel testing approach to facilitate efficient testing and quality assurance. Furthermore, we propose a tool that records the details of testing activity and visualizes them in multiple ways. Our approach realizes both high efficiency like exploratory testing and high auditability like scripted testing. We interviewed 14 developers about our approach to refine it and make it practical. We describe how we reflected the feedback from the interview in our approach.

Test automation tools such as Selenium are commonly used for automating end-to-end tests, but when developers update the software, they often need to modify the test scripts accordingly. However, the costs of modifying these test scripts are a big obstacle to test automation because of the scripts' fragility. In particular, locators in test scripts are prone to change. Some prior methods tried to repair broken locators by using structural clues, but these approaches usually cannot handle radical changes to page layouts. In this paper, we propose a novel approach called COLOR (correct locator recommender) to support repairing broken locators in accordance with software updates. COLOR uses various properties as clues obtained from screens (i.e., attributes, texts, images, and positions). We examined which properties are reliable for recommending locators by examining changes between two release versions of software, and the reliability is adopted as the weight of a property. Our experimental results obtained from four open source web applications show that COLOR can present the correct locator in first place with a 77% - 93% accuracy and is more robust against page layout changes than structure-based approaches.

Automated program repair (in short, APR) has been attracting much attention. A variety of APR techniques have been proposed, and they have been evaluated with actual bugs in open source software. Currently, the authors are trying to introduce APR techniques to industrial software development (in short, ISD) to reduce development cost drastically. However, at this moment, there are no studies that report evaluations of APR techniques on ISD. In this paper, we report our ongoing application of APR techniques to ISD and discuss some barriers that we found on the application.

It is generally said that we should not perform code changes for multiple tasks in a single commit. Such code changes are called tangled ones. Committing tangled changes is harmful to developers. For example, it is costly to merge a part of tangled changes with other commits. Moreover, the presence of such tangled changes hinders analyzing code repositories. That is because most of the mining software repository approaches are designed under the assumption that every commit includes only changes for a single task. In this paper, we propose a technique which informs developers that they are about to commit tangled changes. The technique also suggests how to split a given commit into multiple commits by using past code changes. The proposed technique allows developers to determine whether they accept the suggestion or commit as it stands. By providing such support to developers, they can avoid committing tangled changes.

Although there is a principle that states a commit should only include changes for a single task, it is not always respected by developers. This means that code repositories often include commits that contain tangled changes. The presence of such tangled changes hinders analyzing code repositories because most mining software repository (MSR) approaches are designed with the assumption that every commit includes only changes for a single task. In this paper, we propose a technique to inform developers that they are in the process of committing tangled changes. The proposed technique utilizes the changes included in the past commits to judge whether a given commit includes tangled changes. If it determines that the proposed commit may include tangled changes, it offers suggestions on how the tangled changes can be split into a set of untangled changes.