Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, Qing Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

The development of Large Language Models (LLMs) enables LLM-based GUI testing to interact with graphical user interfaces by understanding GUI screenshots and generating actions, which are widely applied in industry and academia. However, current approaches test each app in isolation, lacking mechanisms for experience accumulation and reuse. This limitation often causes GUI testing approaches to miss deeper exploration and fail to trigger bug-prone functionalities. To address this, we propose MemoDroid, a three-layer memory mechanism that augments LLM-based GUI testing with the ability to evolve through repeated interaction. MemoDroid designs episodic memory to capture functional-level testing traces, reflective memory to summarize issue patterns and redundant behaviors, and strategic memory to synthesize cross-app exploration strategies. These memory layers are dynamically retrieved and injected into LLM prompts at runtime, enabling the agent to reuse successful behaviors, avoid ineffective actions, and prioritize bug-prone paths. We implement MemoDroid as a lightweight plugin, which can be integrated into existing LLM-based GUI testing approaches. We evaluate MemoDroid on real-world apps from 15 diverse app categories. Results show that MemoDroid enhances GUI testing performance across five baselines, with activity and code coverage increasing by 79% - 96% and 81% - 97%, and bug detection improving by 57% - 198%. Ablation studies confirm the contributions of each memory layer. Furthermore, MemoDroid detects 49 new bugs in 200 popular apps, with 35 confirmed fixes and 14 acknowledged by developers, showing its practical value in memory-driven GUI testing.

Automatic API recommendation can accelerate developers’ programming and has been studied for years. There are two orthogonal lines of approaches for this task, i.e., information retrieval-based (IR-based) approaches and sequence to sequence (seq2seq) model-based approaches. Although these approaches were reported to have remarkable performance, our observation finds two major drawbacks, i.e., IR-based approaches lack the consideration of relations among the recommended APIs, and seq2seq models do not model the API’s semantic meaning. To alleviate the above two problems, we propose APIGens, which is a retrieval-enhanced large language model (LLM)-based API recommendation approach to recommend an API sequence for a natural language query. The approach first retrieves similar programming questions in history based on the input natural language query, and then scores the results based on API documents via a scorer model. Finally, these results are used as samples for few-shot learning of LLM. To reduce the risk of encountering local optima, we also extract API seeds from the retrieved results to increase the search scope during the LLM generation process. The results show that our approach can achieve 48.41% ROUGE@10 on API sequence recommendation and the 82.61% MAP on API set recommendation, largely outperforming the state-of-the-art baselines.

Internet technology has given rise to an open-collaborative software development paradigm, necessitating the open-collaborative schema to software testing. It enables diverse and globally distributed contributions, but also presents significant challenges to efficient testing processes, coordination among personnel, and management of testing artifacts. At the same time, advancements in AI have enhanced testing capabilities and enabling automation, while also introducing new testing needs and unique challenges for AI-based systems. In this context, this article explores software testing in the open-collaborative and AI-powered era, focusing on the interrelated dimensions of process, personnel, and technology. Among them, process involves managing testing workflows and artifacts to improve efficiency, personnel emphasizes the role of individuals in ensuring testing quality through collaboration and contributions, while technology refers to AI methods that enhance testing capabilities and address challenges in AI-based systems. Furthermore, we delve into the challenges and opportunities arising from emerging technologies such as Large Language Models (LLMs) and the AI model-centric development paradigm.

Crash reports play a crucial role in software maintenance as they inform developers about the issues encountered in mobile applications. Developers must reproduce the reported crash before fixing it, which is extremely time-consuming and tedious. Existing studies have focused on automatic crash reproduction with step-by-step instructions. However, a non-neglectable portion of crash reports only provides a one-sentence overview, which merely describes the final crash-triggering action. These reports require developers to invest more effort in understanding and fixing the issues while existing techniques cannot handle them due to the lack of step-by-step guidance, thus calling for a greater need for automatic support. Leveraging the capability of Large Language Models (LLMs) in combining acting and reasoning, we propose ReActDroid, an automated approach to reproduce mobile application crashes directly from the crash overview. ReActDroid utilizes ReAct prompting to augment the app-specific knowledge and exploration history, enabling the LLM to derive the necessary steps for triggering the crash from a comprehensive and historical perspective. We evaluate ReActDroid on 102 crash reports from 69 popular Android apps and successfully reproduce 57.8% of the crashes, surpassing the performance of state-of-the-art baselines by 69% to 321%. Besides, the average reproducing time is 51.8 seconds, outperforming the baselines by 73% to 949%. We also evaluate the usefulness of ReActDroid with promising results.

Mobile app GUI (Graphical User Interface) pages now contain rich visual information, with the visual semantics of each page helping users understand the application logic. However, these complex visual and functional logics present new challenges to software testing. Existing automated GUI testing methods, constrained by the lack of reliable testing oracles, are limited to detecting crash bugs with obvious abnormal signals. Consequently, many non-crash functional bugs, ranging from unexpected behaviors to logical errors, often evade detection by current techniques. While these non-crash functional bugs can exhibit visual cues that serve as potential testing oracles, they often entail a sequence of screenshots, and detecting them necessitates an understanding of the operational logic among GUI page transitions, which is challenging traditional techniques. Considering the remarkable performance of Multimodal Large Language Models (MLLM) in visual and language understanding, this paper proposesVisionDroid, a novel vision-driven, multi-agent collaborative automated GUI testing approach for detecting non-crash functional bugs. It comprises three agents: Explorer, Monitor, and Detector, to guide the exploration, oversee the testing progress, and spot issues.We also address several challenges,i.e., aligning visual and textual information for MLLM input, achieving functionality-oriented exploration, and inferring test oracles for non-crash bugs, to enhance the performance of functionality bug detection. We evaluateVisionDroidon 590 non-crash bugs and compare it with 12 baselines, it can achieve more than 14%-112% and 108%-147% boost in average recall and precision compared with the best baseline. The ablation study further proves the contribution of each module. Moreover,VisionDroididentifies 43 unknown bugs on Google Play, of which 31 have been fixed.

Mobile apps have become indispensable for accessing and participating in various environments, especially for low-vision users. Users with visual impairments can use screen readers to read the content of each screen and understand the content that needs to be operated. Screen readers need to read the hint-text attribute in the text input component to remind visually impaired users what to fill in. Unfortunately, based on our analysis of 4,501 Android apps with text inputs, over 76% of them are missing hint-text. These issues are mostly caused by developers’ lack of awareness when considering visually impaired individuals. To overcome these challenges, we developed an LLM-based hint-text generation model called HintDroid, which analyzes the GUI information of input components and uses in-context learning to generate the hint-text. To ensure the quality of hint-text generation, we further designed a feedback-based inspection mechanism to further adjust hint-text. The automated experiments demonstrate the high BLEU and a user study further confirms its usefulness. HintDroid can not only help visually impaired individuals, but also help ordinary people understand the requirements of input components. HintDroid demo video: https://youtu.be/FWgfcctRbfI.

Automated Graphical User Interface (GUI) testing plays a crucial role in ensuring app quality, especially as mobile applications have become an integral part of our daily lives. Despite the growing popularity of learning-based techniques in automated GUI testing due to their ability to generate human-like interactions, they still suffer from several limitations, such as low testing coverage, inadequate generalization capabilities, and heavy reliance on training data. Inspired by the success of Large Language Models (LLMs) like ChatGPT in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within this framework, we have also introduced a functionality-aware memory prompting mechanism that equips the LLM with the ability to retain testing knowledge of the whole process and conduct long-term, functionality-based reasoning to guide exploration. We evaluate it on 93 apps from Google Play and demonstrate that it outperforms the best baseline by 32% in activity coverage, and detects 31% more bugs at a faster rate. Moreover, GPTDroid identifies 53 new bugs on Google Play, of which 35 have been confirmed and fixed.

Mobile applications have become a ubiquitous part of our daily life, providing users with access to various services and utilities. Text input, as an important interaction channel between users and applications, plays an important role in core functionality such as search queries, authentication, messaging, etc. However, certain special text (e.g., -18 for Font Size) can cause the app to crash, and generating diversified unusual inputs for fully testing the app is highly demanded. Nevertheless, this is also challenging due to the combination of explosion dilemma, high context sensitivity, and complex constraint relations. This paper proposes InputBlaster which leverages the LLM to automatically generate unusual text inputs for mobile app crash detection. It formulates the unusual inputs generation problem as a task of producing a set of test generators, each of which can yield a batch of unusual text inputs under the same mutation rule. In detail, InputBlaster leverages LLM to produce the test generators together with the mutation rules serving as the reasoning chain, and utilizes the in-context learning schema to demonstrate the LLM with examples for boosting the performance. InputBlaster is evaluated on 36 text input widgets with cash bugs involving 31 popular Android apps, and results show that it achieves 78% bug detection rate, with 136% higher than the best baseline. Besides, we integrate it with the automated GUI testing tool and detect 37 unseen crashes in real-world apps.

Crash reports are vital for software maintenance since they allow the developers to be informed of the problems encountered in the mobile application. Before fixing, developers need to reproduce the crash, which is an extremely time-consuming and tedious task. Existing studies conducted the automatic crash reproduction with the natural language described reproducing steps. Yet we find a non-neglectable portion of crash reports only contain the stack trace when the crash occurs. Such stack-trace-only crashes merely reveal the last GUI page when the crash occurs, and lack step-by-step guidance. Developers tend to spend more effort in understanding the problem and reproducing the crash, and existing techniques cannot work on this, thus calling for a greater need for automatic support. This paper proposes an approach named CrashTranslator to automatically reproduce mobile application crashes directly from the stack trace. It accomplishes this by leveraging a pre-trained Large Language Model to predict the exploration steps for triggering the crash, and designing a reinforcement learning based technique to mitigate the inaccurate prediction and guide the search holistically. We evaluate CrashTranslator on 75 crash reports involving 58 popular Android apps, and it successfully reproduces 61.3% of the crashes, outperforming the state-of-the-art baselines by 109% to 206%. Besides, the average reproducing time is 68.7 seconds, outperforming the baselines by 302% to 1611%. We also evaluate the usefulness of CrashTranslator with promising results.

Pre-trained large language models (LLMs) have recently emerged as a breakthrough technology in natural language processing and artificial intelligence, with the ability to handle large-scale datasets and exhibit remarkable performance across a wide range of tasks. Meanwhile, software testing is a crucial undertaking that serves as a cornerstone for ensuring the quality and reliability of software products. As the scope and complexity of software systems continue to grow, the need for more effective software testing techniques becomes increasingly urgent, making it an area ripe for innovative approaches such as the use of LLMs. This paper provides a comprehensive review of the utilization of LLMs in software testing. It analyzes 102 relevant studies that have used LLMs for software testing, from both the software testing and LLMs perspectives. The paper presents a detailed discussion of the software testing tasks for which LLMs are commonly used, among which test case preparation and program repair are the most representative. It also analyzes the commonly used LLMs, the types of prompt engineering that are employed, as well as the accompanied techniques with these LLMs. It also summarizes the key challenges and potential opportunities in this direction. This work can serve as a roadmap for future research in this area, highlighting potential avenues for exploration, and identifying gaps in our current understanding of the use of LLMs in software testing.

Bug reports are vital for software maintenance that allow the developers being informed of the problems encountered in the software. Before bug fixing, developers need to reproduce the bugs which is an extremely time-consuming and tedious task, and it is highly expected to automate this process. However, it is challenging to do so considering the imprecise or incomplete natural language described in reproducing steps, and the missing or ambiguous single source of information in GUI components. In this paper, we propose a context-aware bug reproduction approach ScopeDroid which automatically reproduces crashes from textual bug reports for mobile apps. It first constructs a state transition graph (STG) and extracts the contextual information of components. We then design a multi-modal neural matching network to derive the fuzzy matching matrix between all candidate GUI events and reproducing steps. With the STG and matching information, it plans the exploration path for reproducing the bug, and enriches the initial STG iteratively. We evaluate the approach on 102 bug reports from 69 popular Android apps, and it successfully reproduces 63.7% of the crashes, outper-forming the state-of-the-art baselines by 32.6% and 38.3%. We also evaluate the usefulness and robustness of ScopeDroid with promising results. Furthermore, to train the neural matching network, we develop a heuristic-based automated training data generation method, which can potentially motivate and facilitate other activities as user interface operations.

Automated GUI testing is widely used to help ensure the quality of mobile apps. However, many GUIs require appropriate text inputs to proceed to the next page, which remains a prominent obstacle for testing coverage. Considering the diversity and semantic requirement of valid inputs (e.g., flight departure, movie name), it is challenging to automate the text input generation. Inspired by the fact that the pre-trained Large Language Model (LLM) has made outstanding progress in text generation, we propose an approach named QTypist based on LLM for intelligently generating semantic input text according to the GUI context. To boost the performance of LLM in the mobile testing scenario, we develop a prompt-based data construction and tuning method which automatically extracts the prompts and answers for model tuning. We evaluate QTypist on 106 apps from Google Play, and the result shows that the passing rate of QTypist is 87%, which is 93% higher than the best baseline. We also integrate QTypist with the automated GUI testing tools and it can cover 42% more app activities, 52% more pages, and subsequently help reveal 122% more bugs compared with the raw tool.

Mobile apps are indispensable for people's daily life. With the increase of GUI functions, apps have become more complex and diverse. As the Android app is event-driven, Activity Transition Graph (ATG) becomes an important way of app abstract and graphical user interface (GUI) modeling. Although existing works provide static and dynamic analysis to build ATG for applications, the completeness of ATG obtained is poor due to the low coverage of these techniques. To tackle this challenge, we propose a novel approach, ArchiDroid, to automatically augment the ATG via graph convolution network. It models both the semantics of activities and the graph structure of activity transitions to predict the transition between activities based on the seed ATG extracted by static analysis. The evaluation demonstrates that ArchiDroid can achieve 86% precision and 94% recall in predicting the transition between activities for augmenting ATG. We further apply the augmented ATG in two downstream tasks, i.e., guidance in automated GUI testing and assistance in app function design. Results show that the automated GUI testing tool integrated with ArchiDroid achieves 43% more activity coverage and detects 208% more bugs. Besides, ArchiDroid can predict the missing transition with 85% accuracy in real-world apps for assisting the app function design, and an interview case study further demonstrates its usefulness.

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the upgrading of mobile devices and the development of aesthetics, the visual effects of the GUI are more and more attracting, and users pay more attention to the accessibility and usability of applications. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, component occlusion, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a fully automated approach,Nighthawk, based on deep learning for modelling visual information of the GUI screenshot.Nighthawkcan detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. At the same time, training the model needs a large amount of labeled buggy screenshots, which requires considerable manual effort to prepare them. We therefore propose a heuristic-based training data auto-generation method to automatically generate the labeled training data. The evaluation demonstrates that ourNighthawkcan achieve average 0.84 precision and 0.84 recall in detecting UI display issues, average 0.59 AP and 0.60 AR in localizing these issues. We also evaluateNighthawkwith popular Android apps on Google Play and F-Droid, and successfully uncover 151 previously-undetected UI display issues with 75 of them being confirmed or fixed so far.

Mobile apps are indispensable for people’s daily life. Complementing with automated GUI testing, manual testing is the last line of defence for app quality. However, the repeated actions and easily missing of functionalities make manual testing time-consuming and inefficient. Inspired by the game candy crush with flashy candies as hint moves for players, we propose an approach named NaviDroid for navigating testers via highlighted next operations for more effective and efficient testing. Within NaviDroid, we construct an enriched state transition graph with the triggering actions as the edges for two involved states. Based on it, we utilize the dynamic programming algorithm to plan the exploration path, and augment the GUI with visualized hints for testers to quickly explore untested activities and avoid duplicate explorations. The automated experiments demonstrate the high coverage and efficient path planning of NaviDroid and a user study further confirms its usefulness. The NaviDroid can help us develop more robust software that works in more mission-critical settings, not only by performing more thorough testing with the same effort that has been put in before, but also by integrating these techniques into different parts of development pipeline.

As the bridge between users and software, Graphical User Interface (GUI) is critical to the app accessibility. Scaling up the font or display size of GUI can help improve the visual impact, readability, and usability of an app, and is frequently used by the elderly and people with vision impairment. Yet this can easily lead to scaling issues such as text truncation, component overlap, which negatively influence the acquirement of the right information and the fluent usage of the app. Previous techniques for UI display issue detection and cross-platform inconsistency detection cannot work well for these scaling issues. In this paper, we propose an automated method, dVermin, for scaling issue detection, through detecting the inconsistency of a view under the default and a larger display scale. The evaluation result shows that dVermin achieves 97% precision and 97% recall in issue page detection, and 84% precision and 91% recall for issue view detection, outperforming two state-of-the-art baselines by a large margin. We also evaluate dVermin with popular Android apps on F-droid, and successfully uncover 21 previously-undetected scaling issues with 20 of them being confirmed/fixed.

Background. Where to start contributing to a project is a critical challenge for newcomers of open source projects. To support newcomers, GitHub utilizes the Good First Issue (GFI) label, with which project members can manually tag issues in an open source project that are suitable for the newcomers. However, manually labeling GFIs is time- and effort-consuming given the large number of candidate issues. In addition, project members need to have a close understanding of the project to label GFIs accurately.

Graphical User Interface (GUI) provides visual bridges between software apps and end users. However, due to the compatibility of software or hardware, UI display issues such as text overlap, blurred screen, image missing always occur during GUI rendering on different devices. Because these UI display issues can be found directly by human eyes, in this paper, we implement an online UI display issue detection tool OwlEyes-Online, which provides a simple and easy-to-use platform for users to realize the automatic detection and localization of UI display issues. The OwlEyes-Online can automatically run the app and get its screenshots and XML files, and then detect the existence of issues by analyzing the screenshots. In addition, OwlEyes-Online can also find the detailed area of the issue in the given screenshots to further remind developers. Finally, OwlEyes-Online will automatically generate test reports with UI display issues detected in app screenshots and send them to users. The OwlEyes-Online was evaluated and proved to be able to accurately detect UI display issues. Tool Link: http://www.owleyes.online:7476 Github Link: https://github.com/franklinbill/owleyes Demo Video Link: https://youtu.be/002nHZBxtCY

Background: Given the invisibility and unpredictability of distributed crowdtesting processes, there is a large number of duplicate reports, and detecting these duplicate reports is an important task to help save testing effort. Although, many approaches have been proposed to automatically detect the duplicates, the comparison among them and the practical guidelines to adopt these approaches in crowdtesting remain vague.

GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on difference devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues. We develop a heuristics-based data augmentation method and a GAN-based data augmentation method for boosting the performance of our OwlEye. At present, the evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues.

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our OwlEye. The evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate OwlEye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.