User interface (UI) display issues, such as widgets occlusion, missing elements, and screen overflow, are emerging as a non-negligible source of user complaints in commercial mobile apps. However, existing automated testing tools typically rely on a vast amount of high-quality training data, making them cost-ineffective for industrial practice. Given that display issues are intuitively recognizable by humans, their diverse appearances can be abstracted by the violation of human commonsense expectations of UI appearance. Therefore, this paper proposes to reduce data requirements in display issue detection through commonsense simulation. Although leveraging large vision-language models (VLMs) to replicate human visual ability looks straightforward, off-the-shelf VLMs lack task-specific knowledge of UI designs and display correctness. To address this, we fine-tune a VLM to learn what constitutes an expected display and to reason potential display issues. This approach is termed as Minuku, an industrial data-efficient UI display issue detector. We evaluate the design effectiveness of Minuku via a set of ablation experiments. Moreover, real-world deployments in one of the largest E-commerce app providers further demonstrate that Minuku can effectively detect 40 previously unknown UI display issues and significantly reduce manual effort in industrial settings.

Redundant test cases, although well-studied in software engineering, are previously underexplored in UI testing of mobile apps. Our study of real-world test suites shows that, equipped with large-scale testing suites, redundancy in UI testing often manifests as redundant UI interactions. Although negligible in traditional script-based workflows, such redundancy severely impacts the efficiency of emerging Large Language Model (LLM)-based UI agents, which incur substantial decision latency and token costs from repeated LLM queries for the same interactions. To this end, based on the idea of reusing LLMs’ former decisions, we present TestWeaver, a cost-effective LLM-based testing framework. Leveraging a semantic annotated UI Transition Graph (UTG), TestWeaver is capable of detecting shared interactions across test cases. It processes each interaction with a single LLM query, and reuses the result whenever the same interaction occurs. We evaluate TestWeaver on real-world test suites from Meituan. It achieves a 92% success rate with an average cost of $0.11 and 89.7 seconds per case, outperforming the state-of-the-art. We have also deployed TestWeaver in a real-world testing workflow at Meituan for over six months. TestWeaver has executed nearly 2,000 test cases and uncovered 10 previously undetected bugs, while reducing manual testing effort by 75%.

In industrial practice, GUI (Graphic User Interface) testing of mobile apps still inevitably relies on huge manual efforts. The major efforts are those on understanding the GUIs, so that testing scripts can be written accordingly. Quality assurance could therefore be very labor-intensive, especially for modern commercial mobile apps, where one may include tremendous, diverse, and complex GUIs, e.g., those for placing orders of different commercial items. To reduce such human efforts, we propose Appaction, a learning-based automatic GUI interaction approach we developed for Meituan, one of the largest E-commerce providers with over 600 million users. Appaction can automatically analyze the target GUI and understand what each input of the GUI is about, so that corresponding valid inputs can be entered accordingly. To this end, Appaction adopts a multi-modal model to learn from human experiences in perceiving a GUI. This allows it to infer corresponding valid input events that can properly interact with the GUI. In this way, the target app can be effectively exercised. We present our experiences in Meituan on applying Appaction to popular commercial apps. We demonstrate the effectiveness of Appaction in GUI analysis, and it can perform correct interactions for numerous form pages.