The openness of crowdsourced testing introduces diversity in testing results. However, it also leads to a large volume of test reports, many of which highlight the same recurring issues. While these reports provide valuable feedback, their redundancy makes it inefficient for developers to review the reports and identify bugs. Crowdsourced test report clustering has been proposed to mitigate this problem, allowing developers to focus only on the representative reports from each cluster. However, existing methods primarily rely on embedding features extracted from reports for clustering, which limits their ability to generate accurate and interpretable clusters due to a lack of deeper semantic understanding of the reports. To address the aforementioned challenge, we propose LLMCluster , a novel method for crowdsourced test report clustering based on Large Language Models (LLMs). LLMCluster employs an iterative clustering strategy. In each iteration, LLMCluster processes a subset of reports by instructing the LLM to disregard surface-level variations in expression, analyze the core issue in each report, and group reports addressing the same issue into new or existing clusters. After the iterative clustering process, LLMCluster applies correction algorithms to ensure the completeness and validity of the clustering result. Finally, LLMCluster utilizes the LLM to generate concise summaries for each cluster, making the results more intuitive and interpretable. Experimental results show that LLMCluster outperforms state-of-the-art methods across six commonly used clustering evaluation metrics. Additionally, the cluster summaries generated by LLMCluster semantically align well with manually written summaries.

Crowdsourced testing has emerged as a prominent paradigm in software testing by leveraging the diversity of crowdworkers. In this paradigm, crowd-workers are required to submit a test report for each identified bug, which typically contains a textual description and a bug screenshot. However, due to varying worker expertise, many reports exhibit inconsistencies between the textual description and the bug screenshot, which hinder the report review process. Existing methods address this issue by automatically detecting report consistency, typically through matching the UI widgets referenced in the textual description with those visible in the bug screenshot. However, such methods focus only on surface-level element correspondence and fail to capture the abstract bug semantics, such as the functional meaning and bug-triggering context. Consequently, they lack the ability to detect more subtle but realistic inconsistencies. To bridge this gap, we propose INCONHUNTER, a novel method for crowdsourced test report consistency detection that explicitly simulates human cognitive pattern. In this pattern, humans typically adopt two complementary reasoning strategies. If the textual description allows them to form an expectation about the visual bug features, they assess consistency by verifying if the expected features appear in the bug screenshot. Otherwise, they shift to reasoning about if the bug-triggering context described in the report aligns with the app state shown in the bug screenshot. INCONHUNTERinstantiates this cognitive pattern through two LLM-powered modules, each dedicated to one reasoning strategy. We evaluate INCONHUNTERthrough experiments on our dataset with 2,310 labeled crowdsourced test reports, and results show that INCONHUNTERoutperforms baselines by 14.00%–19.28%, demonstrating superior effectiveness, monetary-based cost efficiency, and alignment with human cognitive pattern.

Graphical User Interface (GUI) testing has been a significant topic in the software engineering community. Most existing GUI testing frameworks are intrusive and can only support some specific platforms, which are quite limited. With the development of distinct scenarios, diverse embedded systems or customized operating systems on different devices do not support existing intrusive GUI testing frameworks. Some approaches adopt robotic arms to replace the interface invoking of mobile apps under test and use computer vision technologies to identify GUI elements. However, some challenges remain unsolved with such approaches. First, existing approaches assume that GUI screens are fixed so that they cannot be adapted to diverse systems with different screen conditions. Second, existing approaches use XY-plane robotic arm system, which cannot flexibly simulate human testing operations. Third, existing approaches ignore the compatibility bugs of apps and only focus on the crash bugs. To sum up, a more practical approach is required for the non-intrusive scenario.

Software applications (apps) have been playing an increasingly important role in various aspects of society. In particular, mobile apps and web apps are the most prevalent among all applications and are widely used in various industries as well as in people’s daily lives. To help ensure mobile and web app quality, many approaches have been introduced to improve app GUI testing via automated exploration, including random testing, model-based testing, learning-based testing, and so on. Despite the extensive effort, existing approaches are still limited in reaching high code coverage, constructing high-quality models, and being generally applicable. Reinforcement learning-based approaches, as a group of representative and advanced approaches for automated GUI exploration testing, are faced with difficult challenges, including effective app state abstraction, reward function design, and so on. Moreover, they heavily depend on the specific execution platforms (i.e., Android or Web), thus leading to poor generalizability and being unable to adapt to different platforms. This work specifically tackles these challenges based on the high-level observation that apps from distinct platforms share commonalities in GUI design. Indeed, we propose PIRLTest , an effective platform-independent approach for app testing. Specifically, PIRLTest utilizes computer vision and reinforcement learning techniques in a novel, synergistic manner for automated testing. It extracts the GUI widgets from GUI pages and characterizes the corresponding GUI layouts, embedding the GUI pages as states. The app GUI state combines the macroscopic perspective (app GUI layout) and the microscopic perspective (app GUI widget) and attaches the critical semantic information from GUI images. This enables PIRLTest to be platform-independent and makes the testing approach generally applicable on different platforms. PIRLTest explores apps with the guidance of a curiosity-driven strategy, which uses a Q-network to estimate the values of specific state-action pairs to encourage more exploration in uncovered pages without platform dependency. The exploration will be assigned with rewards for all actions, which are designed considering both the app GUI states and the concrete widgets, to help the framework explore more uncovered pages. We conduct an empirical study on 20 mobile apps and 5 web apps, and the results show that PIRLTest is zero-cost when being adapted to different platforms, and can perform better than the baselines, covering 6.3–41.4% more code on mobile apps and 1.5–51.1% more code on web apps. PIRLTest is capable of detecting 128 unique bugs on mobile and web apps, including 100 bugs that cannot be detected by the baselines.

This paper investigates the application of large language models (LLM) in the domain of mobile application test script generation. Test script generation is a vital component of software testing, enabling efficient and reliable automation of repetitive test tasks. However, existing generation approaches often encounter limitations, such as difficulties in accurately capturing and reproducing test scripts across diverse devices, platforms, and applications. These challenges arise due to differences in screen sizes, input modalities, platform behaviors, API inconsistencies, and application architectures. Overcoming these limitations is crucial for achieving robust and comprehensive test automation.By leveraging the capabilities of LLMs, we aim to address these challenges and explore its potential as a versatile tool for test automation. We investigate how well LLMs can adapt to diverse devices and systems while accurately capturing and generating test scripts. Additionally, we evaluate its cross-platform generation capabilities by assessing its ability to handle operating system variations and platform-specific behaviors. Furthermore, we explore the application of LLMs in cross-app migration, where it generates test scripts across different applications and software environments based on existing scripts.Throughout the investigation, we analyze its adaptability to various user interfaces, app architectures, and interaction patterns, ensuring accurate script generation and compatibility. The findings of this research contribute to the understanding of LLMs’ capabilities in test automation. Ultimately, this research aims to enhance software testing practices, empowering app developers to achieve higher levels of software quality and development efficiency.