Evaluating security and reliability for multi-agent systems (MAS) is urgent as they become increasingly prevalent in various applications. As an evaluation technique, existing adversarial attack frameworks face certain limitations, e.g., impracticality due to the requirement of white-box information or high control authority, and a lack of stealthiness or effectiveness as they often target all agents or specific fixed agents. To address these issues, we propose AdapAM, a novel framework for adversarial attacks on black-box MAS. AdapAM incorporates two key components: (1) Adaptive Selection Policy simultaneously selects the victim and determines the anticipated malicious action (the action would lead to the worst impact on MAS), balancing effectiveness and stealthiness. (2) Proxy-based Perturbation to Induce Malicious Action utilizes generative adversarial imitation learning to approximate the target MAS, allowing AdapAM to generate perturbed observations using white-box information and thus induce victims to execute malicious action in black-box settings. We evaluate AdapAM across eight multi-agent environments and compare it with four state-of-the-art and commonly-used baselines. Results demonstrate that AdapAM achieves the best attack performance in different perturbation rates. Besides, AdapAM-generated perturbations are the least noisy and hardest to detect, emphasizing the stealthiness.

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by retrieving relevant documents from external corpora before generating responses. This approach significantly expands LLM capabilities by leveraging vast, up-to-date external knowledge. However, this reliance on external knowledge makes RAG systems vulnerable to corpus poisoning attacks that manipulate generated outputs via poisoned document injection. Existing poisoning attack strategies typically treat the retrieval and generation stages as disjointed, limiting their effectiveness. We propose Joint-GCG, the first framework to unify gradient-based attacks across both retriever and generator models through three innovations: (1) Cross-Vocabulary Projection for aligning embedding spaces, (2) Gradient Tokenization Alignment for synchronizing token-level gradient signals, and (3) Adaptive Weighted Fusion for dynamically balancing attacking objectives. Evaluations demonstrate that Joint-GCG achieves at most 25% and an average of 5% higher attack success rate than previous methods across multiple retrievers and generators. While optimized under a white-box assumption, the generated poisons show unprecedented transferability to unseen models. Joint-GCG's innovative unification of gradient-based attacks across retrieval and generation stages fundamentally reshapes our understanding of vulnerabilities within RAG systems.

Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, Qing Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Jie Huang, Xin Liao, Junjie Wang, Mingyang Li, Wenshuo Wang, Ziyou Jiang, Shoubin Li, Qing Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Ziyou Jiang, Mingyang Li, Junjie Wang, Yuekai Huang, Jie Huang, Zhiyuan Chang, Zhaoyang Li, Qing Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large Language Models (LLMs) have emerged as central planners in Vision-and-Language Navigation (VLN), yet their complexity increasingly obscures their internal decision-making. Existing interpretability methods typically isolate temporal criticality from feature salience, creating an alignment gap and failing to account for the behavioral instability of black-box agents. To address this, we propose DEFT, a unified dual-view framework that demystifies agent behavior by jointly analyzing \textit{when} a decision is pivotal and \textit{what} visual evidence grounds it. Featuring a dual-head architecture with a shared latent representation, DEFT employs a \textit{Mask Head} for counterfactual-based criticality detection and an \textit{Action Head} that leverages an ensemble of surrogates to recover robust visual cues. Extensive experiments on MatterPort3D across three LLM-based agents demonstrate that DEFT outperforms baselines in both temporal and feature fidelity. User studies further validate its utility, showing 78% alignment with human intuition.

Visual entailment (VE) is a multimodal reasoning task consisting of image-sentence pairs whereby a promise is defined by an image, and a sentence describes a hypothesis. The goal is to predict whether the image semantically entails the sentence. VE systems have been widely adopted in many downstream tasks such as image caption and visual question answering. However, the robustness of VE systems still faces significant challenges. One of the reasons is that the VE system suffers object-confusing defect when some similar objects exist. It outputs a positive prediction inferred by an erroneous object relationship, which will result in a fault negative prediction if the noised object does not exist. Previous approaches generate tests primarily relied on some general perturbations, such as simulating noise or weather interference in images, or substituting synonyms or rewriting sentences in texts. To test the object-confusing defect in VE systems, it requires perceiving and understanding key objects and entities and maintain the semantic relevance between cross-modal inputs, making it challenging to generate effective tests with high quality. Therefore, we propose VEglue , an object-aligned joint erasing approach for VE systems testing. It first aligns the object regions in the premise and object descriptions in the hypothesis to identify linked and un-linked objects. Then, based on the alignment information, three metamorphic relations are designed to jointly erase the objects of the two modalities. We evaluate VEglue on four widely used VE systems involving two public datasets, and the results demonstrate that VEglue could detect 11,609 issues on average with a 52.5% Issue Finding Rate (IFR). Furthermore, we leverage the tests generated by VEglue to retrain the VE systems, which largely improves model performance (50.8% increase in accuracy) on newly generated tests without sacrificing the accuracy on the original test set.

Explaining multi-agent systems (MAS) is urgent as these systems become increasingly prevalent in various applications. Previous work has provided explanations for the actions or states of agents, yet falls short in understanding the blackboxed agent’s importance within a MAS and the overall team strategy. To bridge this gap, we propose EMAI, a novel agent-level explanation approach that evaluates the individual agent’s importance. Inspired by counterfactual reasoning, a larger change in reward caused by the randomized action of agent indicates its higher importance. We model it as a MARL problem to capture interactions across agents. Utilizing counterfactual reasoning, EMAI learns the masking agents to identify important agents. Specifically, we define the optimization function to minimize the reward difference before and after action randomization and introduce sparsity constraints to encourage the exploration of more action randomization of agents during training. The experimental results in seven multi-agent tasks demonstrate that EMAI achieves higher fidelity in explanations compared to baselines and provides more effective guidance in practical applications concerning understanding policies, launching attacks, and patching policies.

Information theft attacks pose a significant risk to Large Language Model (LLM) tool-learning systems. Adversaries can inject malicious commands through compromised tools, manipulating LLMs to send sensitive information to these tools, which leads to potential privacy breaches. However, existing attack approaches are black-box oriented and rely on static commands that cannot adapt flexibly to the changes in user queries and the invocation chain of tools. It makes malicious commands more likely to be detected by LLM and leads to attack failure. In this paper, we propose AutoCMD, a dynamic attack comment generation approach for information theft attacks in LLM tool-learning systems. Inspired by the concept of mimicking the familiar, AutoCMD is capable of inferring the information utilized by upstream tools in the toolchain through learning on open-source systems and reinforcement with target system examples, thereby generating more targeted commands for information theft. The evaluation results show that AutoCMD outperforms the baselines with +13.2% ASR_{Theft}, and can be generalized to new tool-learning systems to expose their information leakage risks. We also design four defense methods to effectively protect tool-learning systems from the attack.

The development of Large Language Models (LLMs) enables LLM-based GUI testing to interact with graphical user interfaces by understanding GUI screenshots and generating actions, which are widely applied in industry and academia. However, current approaches test each app in isolation, lacking mechanisms for experience accumulation and reuse. This limitation often causes GUI testing approaches to miss deeper exploration and fail to trigger bug-prone functionalities. To address this, we propose MemoDroid, a three-layer memory mechanism that augments LLM-based GUI testing with the ability to evolve through repeated interaction. MemoDroid designs episodic memory to capture functional-level testing traces, reflective memory to summarize issue patterns and redundant behaviors, and strategic memory to synthesize cross-app exploration strategies. These memory layers are dynamically retrieved and injected into LLM prompts at runtime, enabling the agent to reuse successful behaviors, avoid ineffective actions, and prioritize bug-prone paths. We implement MemoDroid as a lightweight plugin, which can be integrated into existing LLM-based GUI testing approaches. We evaluate MemoDroid on real-world apps from 15 diverse app categories. Results show that MemoDroid enhances GUI testing performance across five baselines, with activity and code coverage increasing by 79% - 96% and 81% - 97%, and bug detection improving by 57% - 198%. Ablation studies confirm the contributions of each memory layer. Furthermore, MemoDroid detects 49 new bugs in 200 popular apps, with 35 confirmed fixes and 14 acknowledged by developers, showing its practical value in memory-driven GUI testing.

Rupeng Zhang, Haowei Wang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, Qing Wang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

The competitive game between agents exists in many critical applications, such as military unmanned aerial vehicles. It is urgent to test these agents to reduce the significant losses caused by their failures. Existing studies mainly are to construct a testing agent that competes with the target agent to induce its failures. These approaches usually focus on a single task, requiring much more time for multi-task testing. However, if the previously tested tasks (source tasks) and the task to be tested (target task) share similar agents or task objectives, the transferable knowledge in source tasks can potentially increase the effectiveness of testing in the target task. We propose Demo2Test for conducting transfer testing of agents in the competitive environment, i.e., leveraging the demonstrations of failure scenarios from the source task to boost the testing effectiveness in the target task. It trains a testing agent with demonstrations and incorporates the action perturbation at key states to balance the number of revealed failures and their diversity. We conduct experiments in the simulated robotics competitive environments of MuJoCo. The results indicate that Demo2Test outperforms the best-performing baseline with improvements ranging from \(22.38\%\) to \(87.98\%\) , and \(12.69\%\) to \(60.98\%\) , in terms of the number and diversity of discovered failure scenarios, respectively.

Automatic API recommendation can accelerate developers’ programming and has been studied for years. There are two orthogonal lines of approaches for this task, i.e., information retrieval-based (IR-based) approaches and sequence to sequence (seq2seq) model-based approaches. Although these approaches were reported to have remarkable performance, our observation finds two major drawbacks, i.e., IR-based approaches lack the consideration of relations among the recommended APIs, and seq2seq models do not model the API’s semantic meaning. To alleviate the above two problems, we propose APIGens, which is a retrieval-enhanced large language model (LLM)-based API recommendation approach to recommend an API sequence for a natural language query. The approach first retrieves similar programming questions in history based on the input natural language query, and then scores the results based on API documents via a scorer model. Finally, these results are used as samples for few-shot learning of LLM. To reduce the risk of encountering local optima, we also extract API seeds from the retrieved results to increase the search scope during the LLM generation process. The results show that our approach can achieve 48.41% ROUGE@10 on API sequence recommendation and the 82.61% MAP on API set recommendation, largely outperforming the state-of-the-art baselines.

Internet technology has given rise to an open-collaborative software development paradigm, necessitating the open-collaborative schema to software testing. It enables diverse and globally distributed contributions, but also presents significant challenges to efficient testing processes, coordination among personnel, and management of testing artifacts. At the same time, advancements in AI have enhanced testing capabilities and enabling automation, while also introducing new testing needs and unique challenges for AI-based systems. In this context, this article explores software testing in the open-collaborative and AI-powered era, focusing on the interrelated dimensions of process, personnel, and technology. Among them, process involves managing testing workflows and artifacts to improve efficiency, personnel emphasizes the role of individuals in ensuring testing quality through collaboration and contributions, while technology refers to AI methods that enhance testing capabilities and address challenges in AI-based systems. Furthermore, we delve into the challenges and opportunities arising from emerging technologies such as Large Language Models (LLMs) and the AI model-centric development paradigm.

Crash reports play a crucial role in software maintenance as they inform developers about the issues encountered in mobile applications. Developers must reproduce the reported crash before fixing it, which is extremely time-consuming and tedious. Existing studies have focused on automatic crash reproduction with step-by-step instructions. However, a non-neglectable portion of crash reports only provides a one-sentence overview, which merely describes the final crash-triggering action. These reports require developers to invest more effort in understanding and fixing the issues while existing techniques cannot handle them due to the lack of step-by-step guidance, thus calling for a greater need for automatic support. Leveraging the capability of Large Language Models (LLMs) in combining acting and reasoning, we propose ReActDroid, an automated approach to reproduce mobile application crashes directly from the crash overview. ReActDroid utilizes ReAct prompting to augment the app-specific knowledge and exploration history, enabling the LLM to derive the necessary steps for triggering the crash from a comprehensive and historical perspective. We evaluate ReActDroid on 102 crash reports from 69 popular Android apps and successfully reproduce 57.8% of the crashes, surpassing the performance of state-of-the-art baselines by 69% to 321%. Besides, the average reproducing time is 51.8 seconds, outperforming the baselines by 73% to 949%. We also evaluate the usefulness of ReActDroid with promising results.

Mobile app GUI (Graphical User Interface) pages now contain rich visual information, with the visual semantics of each page helping users understand the application logic. However, these complex visual and functional logics present new challenges to software testing. Existing automated GUI testing methods, constrained by the lack of reliable testing oracles, are limited to detecting crash bugs with obvious abnormal signals. Consequently, many non-crash functional bugs, ranging from unexpected behaviors to logical errors, often evade detection by current techniques. While these non-crash functional bugs can exhibit visual cues that serve as potential testing oracles, they often entail a sequence of screenshots, and detecting them necessitates an understanding of the operational logic among GUI page transitions, which is challenging traditional techniques. Considering the remarkable performance of Multimodal Large Language Models (MLLM) in visual and language understanding, this paper proposesVisionDroid, a novel vision-driven, multi-agent collaborative automated GUI testing approach for detecting non-crash functional bugs. It comprises three agents: Explorer, Monitor, and Detector, to guide the exploration, oversee the testing progress, and spot issues.We also address several challenges,i.e., aligning visual and textual information for MLLM input, achieving functionality-oriented exploration, and inferring test oracles for non-crash bugs, to enhance the performance of functionality bug detection. We evaluateVisionDroidon 590 non-crash bugs and compare it with 12 baselines, it can achieve more than 14%-112% and 108%-147% boost in average recall and precision compared with the best baseline. The ablation study further proves the contribution of each module. Moreover,VisionDroididentifies 43 unknown bugs on Google Play, of which 31 have been fixed.

Deep reinforcement learning has achieved remarkable success in competitive games, surpassing human performance in applications ranging from business competitions to video games. In competitive environments, agents face the challenge of adapting to continuously shifting adversary strategies, necessitating the ability to handle diverse scenarios. Existing studies primarily focus on evaluating agent robustness either through perturbing observations, which has practical limitations, or through training adversarial agents to expose weaknesses, which lacks strategy diversity exploration. There are also studies which rely on curiosity-based mechanism to explore the diversity, yet they may lack direct guidance to enhance identified decision-making flaws. In this paper, we propose a novel diversity-oriented testing framework (called AdvTest) to test the competitive game agent via constraint-guided adversarial agent training. Specifically, AdvTest adds constraints as the explicit guidance during adversarial agent training to make it capable of defeating the target agent using diverse strategies. To realize the method, three challenges need to be addressed, i.e., what are the suitable constraints, when to introduce constraints, and which constraint should be added. We experimentally evaluate AdvTest on the commonly-used competitive game environment, StarCraft II. The results on four maps show that AdvTest exposes more diverse failure scenarios compared with the commonly-used and state-of-the-art baselines.

Mobile apps have become indispensable for accessing and participating in various environments, especially for low-vision users. Users with visual impairments can use screen readers to read the content of each screen and understand the content that needs to be operated. Screen readers need to read the hint-text attribute in the text input component to remind visually impaired users what to fill in. Unfortunately, based on our analysis of 4,501 Android apps with text inputs, over 76% of them are missing hint-text. These issues are mostly caused by developers’ lack of awareness when considering visually impaired individuals. To overcome these challenges, we developed an LLM-based hint-text generation model called HintDroid, which analyzes the GUI information of input components and uses in-context learning to generate the hint-text. To ensure the quality of hint-text generation, we further designed a feedback-based inspection mechanism to further adjust hint-text. The automated experiments demonstrate the high BLEU and a user study further confirms its usefulness. HintDroid can not only help visually impaired individuals, but also help ordinary people understand the requirements of input components. HintDroid demo video: https://youtu.be/FWgfcctRbfI.

Automated Graphical User Interface (GUI) testing plays a crucial role in ensuring app quality, especially as mobile applications have become an integral part of our daily lives. Despite the growing popularity of learning-based techniques in automated GUI testing due to their ability to generate human-like interactions, they still suffer from several limitations, such as low testing coverage, inadequate generalization capabilities, and heavy reliance on training data. Inspired by the success of Large Language Models (LLMs) like ChatGPT in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within this framework, we have also introduced a functionality-aware memory prompting mechanism that equips the LLM with the ability to retain testing knowledge of the whole process and conduct long-term, functionality-based reasoning to guide exploration. We evaluate it on 93 apps from Google Play and demonstrate that it outperforms the best baseline by 32% in activity coverage, and detects 31% more bugs at a faster rate. Moreover, GPTDroid identifies 53 new bugs on Google Play, of which 35 have been confirmed and fixed.

Mobile applications have become a ubiquitous part of our daily life, providing users with access to various services and utilities. Text input, as an important interaction channel between users and applications, plays an important role in core functionality such as search queries, authentication, messaging, etc. However, certain special text (e.g., -18 for Font Size) can cause the app to crash, and generating diversified unusual inputs for fully testing the app is highly demanded. Nevertheless, this is also challenging due to the combination of explosion dilemma, high context sensitivity, and complex constraint relations. This paper proposes InputBlaster which leverages the LLM to automatically generate unusual text inputs for mobile app crash detection. It formulates the unusual inputs generation problem as a task of producing a set of test generators, each of which can yield a batch of unusual text inputs under the same mutation rule. In detail, InputBlaster leverages LLM to produce the test generators together with the mutation rules serving as the reasoning chain, and utilizes the in-context learning schema to demonstrate the LLM with examples for boosting the performance. InputBlaster is evaluated on 36 text input widgets with cash bugs involving 31 popular Android apps, and results show that it achieves 78% bug detection rate, with 136% higher than the best baseline. Besides, we integrate it with the automated GUI testing tool and detect 37 unseen crashes in real-world apps.

Crash reports are vital for software maintenance since they allow the developers to be informed of the problems encountered in the mobile application. Before fixing, developers need to reproduce the crash, which is an extremely time-consuming and tedious task. Existing studies conducted the automatic crash reproduction with the natural language described reproducing steps. Yet we find a non-neglectable portion of crash reports only contain the stack trace when the crash occurs. Such stack-trace-only crashes merely reveal the last GUI page when the crash occurs, and lack step-by-step guidance. Developers tend to spend more effort in understanding the problem and reproducing the crash, and existing techniques cannot work on this, thus calling for a greater need for automatic support. This paper proposes an approach named CrashTranslator to automatically reproduce mobile application crashes directly from the stack trace. It accomplishes this by leveraging a pre-trained Large Language Model to predict the exploration steps for triggering the crash, and designing a reinforcement learning based technique to mitigate the inaccurate prediction and guide the search holistically. We evaluate CrashTranslator on 75 crash reports involving 58 popular Android apps, and it successfully reproduces 61.3% of the crashes, outperforming the state-of-the-art baselines by 109% to 206%. Besides, the average reproducing time is 68.7 seconds, outperforming the baselines by 302% to 1611%. We also evaluate the usefulness of CrashTranslator with promising results.

Multi-agent systems (MASs) have achieved remarkable success in multi-robot control, intelligent transportation, and multiplayer games, etc. Thorough testing for MAS is urgently needed to ensure its robustness in the face of constantly changing and unexpected scenarios. Existing methods mainly focus on single-agent system testing and cannot be directly applied to MAS testing due to the complexity of MAS. To our best knowledge, there are fewer studies on MAS testing. While several studies have focused on adversarial attacks on MASs, they primarily target failure detection from an attack perspective, i.e., discovering failure scenarios, while ignoring the diversity of scenarios. In this paper, to highlight a typical balance between exploration (diversifying behaviors) and exploitation (detecting failures), we propose an advanced testing framework for MAS called with diversity-guided exploration and adaptive critical state exploitation. It incorporates both individual diversity and team diversity, and designs an adaptive perturbation mechanism to perturb the action at the critical states, so as to trigger more and more diverse failure scenarios of the system. We evaluate MASTest on two popular MAS simulation environments: Coop Navi and StarCraft II. Results show that the average distance of the resulting failure scenarios is increased by 29.55%-103.57% and 74.07%-370.00% on two environments compared to the baselines. Also, the failure patterns found by MASTest are improved by 71.44%-300.00% and 50%-500.00% on two experimental environments compared to the baselines.

Security patches are essential for enhancing the stability and robustness of projects in the open-source software community. While vulnerabilities are officially expected to be patched before being disclosed, patching vulnerabilities is complicated and remains a struggle for many organizations. To patch vulnerabilities, security practitioners typically track vulnerable issue reports (IRs), and analyze their relevant insecure code to generate potential patches. However, the relevant insecure code may not be explicitly specified and practitioners cannot track the insecure code in the repositories, thus limiting their ability to generate patches. In such cases, providing examples of insecure code and the corresponding patches would benefit the security developers to better locate and resolve the actual insecure code. In this paper, we propose PatUntrack, an automated approach to generating patch examples from IRs without tracked insecure code. PatUntrack utilizes auto-prompting to optimize the Large Language Model (LLM) to make it applicable for analyzing the vulnerabilities described in IRs and generating appropriate patch examples. Specifically, it first generates the completed description of the Vulnerability-Triggering Path (VTP) from vulnerable IRs. Then, it corrects potential hallucinations in the VTP description with external golden knowledge. Finally, it generates Top-K pairs of Insecure Code and Patch Example based on the corrected VTP description. To evaluate the performance of PatUntrack, we conducted experiments on 5,465 vulnerable IRs. The experimental results show that PatUntrack can obtain the highest performance and improve the traditional LLM baselines by +17.7% (MatchFix) and +14.6% (Fix@10) on average in patch example generation. Furthermore, PatUntrack was applied to generate patch examples for 76 newly disclosed vulnerable IRs. 27 out of 37 replies from the authors of these IRs confirmed the usefulness of the patch examples generated by PatUntrack, indicating that they can benefit from these examples for patching the vulnerabilities.

Inefficient API usage is one of the main reasons for software performance issues. Current practice of API documentation mainly provides its functionalities, while the performance related information are seldom covered in the official documentation. Meanwhile, the online discussions brings various pieces of information about the efficiency of API, yet buried in massive messages. Existing approaches would derive API opinion with pattern-based techniques, and typically result in inaccurate and coarse-grained result. This paper proposes a relation-aware approach RAMiner for the fine-grained API-related performance opinion mining from online discussions. It leverages pre-trained Large Language Model (LLM), thus can better capture the semantics of the text and API tokens. Besides, it disentangles the task into subtasks to cope with the situation of limited labeled data for fine-tuning the model, and incorporates relation-aware design for capturing the fine-grained opinion of each mentioned API. The experimental results show that, RAMiner can correctly predict 70% opinions, which largely outperforms the baselines. We also demonstrate its potential usage in promoting the code generation models in recommending more efficient code snippets. This approach can also be utilized to extract other non-functional opinions, e.g., security, compatibility.

Recently, a growing body of research has realized that live chat via modern communication platforms plays an increasingly important role in OSS (Open Source Software) collaborative development. Among these platforms, Gitter has emerged as a popular choice since it is directed toward GitHub projects by account sharing and activity subscribing. But little is known about how Gitter affects the OSS development on GitHub. Who are the developers being active in both social and technical platforms? How important are they? In this paper, we perform a comprehensive cross-platform study on Gitter and GitHub, two representative platforms for live communication and distributed development, to explore the characteristics of cross-platform contributors (CPCs) and whether live chat can provoke open source development. This study yields interesting findings: 1) Despite CPCs being small in quantity yet account for a much bigger amount of communication and development; 2) Gitter continually attracts new contributors; 3) Communication on Gitter has a positive impact on the contributions of OSS developers; and 4) Inactive developers on GitHub still participate in discussions on Gitter. Based on our findings, we provide recommendations for OSS communities and developers and shed light on future research directions. We believe that the findings and insights will inspire the OSS communities, enable a broader view of the interplay between Gitter and GitHub, and enhance the sustainability of the OSS ecosystem.

Pre-trained large language models (LLMs) have recently emerged as a breakthrough technology in natural language processing and artificial intelligence, with the ability to handle large-scale datasets and exhibit remarkable performance across a wide range of tasks. Meanwhile, software testing is a crucial undertaking that serves as a cornerstone for ensuring the quality and reliability of software products. As the scope and complexity of software systems continue to grow, the need for more effective software testing techniques becomes increasingly urgent, making it an area ripe for innovative approaches such as the use of LLMs. This paper provides a comprehensive review of the utilization of LLMs in software testing. It analyzes 102 relevant studies that have used LLMs for software testing, from both the software testing and LLMs perspectives. The paper presents a detailed discussion of the software testing tasks for which LLMs are commonly used, among which test case preparation and program repair are the most representative. It also analyzes the commonly used LLMs, the types of prompt engineering that are employed, as well as the accompanied techniques with these LLMs. It also summarizes the key challenges and potential opportunities in this direction. This work can serve as a roadmap for future research in this area, highlighting potential avenues for exploration, and identifying gaps in our current understanding of the use of LLMs in software testing.

BACKGROUND: In an online issue tracking system, e.g., GitHub Issue Tracker, feature requests and the associated comment stream provide valuable crowd-generated knowledge for requirements elicitation. To decide whether a feature request should be accepted or not, stakeholders need to identify the comments for/against the feature and understand the two-sided opinions, which is time- and effort-consuming considering the abundant information embedded in lengthy comment stream per feature request. AIMS: This paper proposes VoteBot for automatically detecting stance (for/against) and summarizing the related opinions on a feature request, which can facilitate the decision making (i.e., voting) of feature requests. To our best knowledge, such an approach is previously unexplored for crowd-based requirements elicitation. METHOD: VoteBot is a relation-aware approach, which incorporates three types of relations among the comments or among the comment sentences to better understand the discussions about feature requests. Specifically, it extracts the reply-to relation among the comments, and incorporates it into a BERT-based classifier for stance detection. It also designs a graph-based ranking algorithm, and incorporates semantic relevance and argumentative relations for stance summarization. RESULTS: The automatic evaluation on 250 feature requests with 6,598 comments from five GitHub projects, and the evaluation with practitioners on five new projects, show the promising results. CONCLUSIONS: VoteBot is effective in stance detection and stance summarization, and potentially useful for understanding feature requests and associated discussions in real-world practice.

Requirements linking is the core of software system maintenance and evolution, and it is critical to assuring software quality. In practice, however, the requirements links are frequently absent or incorrectly labeled, and reconstructing such ties is time-consuming and error-prone. Numerous learning-based approaches have been put forth to address the problem. However, these approaches will lose effectiveness for the Cold-Start projects with few labeled samples. To this end, we propose RADIATION, an adversarial-based domain adaptation approach for cross-domain requirements linking. Generally, RADIATION firstly adopts an IDF-based Masking strategy to filter the domain-specific features. Then it pre-trains a linking model in the source domain with sufficient labeled samples and adapts the model to target domains using a distance-enhanced adversarial technique without using any labeled target samples. Evaluation on five public datasets shows that RADIATION could achieve 66.4% precision, 89.2% recall, and significantly outperform state-of-the-art baselines by 13.4% -42.9% F1. In addition, the designed components, i.e., IDF-based Masking and Distance-enhanced Loss, could significantly improve performance.

Bug reports are vital for software maintenance that allow the developers being informed of the problems encountered in the software. Before bug fixing, developers need to reproduce the bugs which is an extremely time-consuming and tedious task, and it is highly expected to automate this process. However, it is challenging to do so considering the imprecise or incomplete natural language described in reproducing steps, and the missing or ambiguous single source of information in GUI components. In this paper, we propose a context-aware bug reproduction approach ScopeDroid which automatically reproduces crashes from textual bug reports for mobile apps. It first constructs a state transition graph (STG) and extracts the contextual information of components. We then design a multi-modal neural matching network to derive the fuzzy matching matrix between all candidate GUI events and reproducing steps. With the STG and matching information, it plans the exploration path for reproducing the bug, and enriches the initial STG iteratively. We evaluate the approach on 102 bug reports from 69 popular Android apps, and it successfully reproduces 63.7% of the crashes, outper-forming the state-of-the-art baselines by 32.6% and 38.3%. We also evaluate the usefulness and robustness of ScopeDroid with promising results. Furthermore, to train the neural matching network, we develop a heuristic-based automated training data generation method, which can potentially motivate and facilitate other activities as user interface operations.

Automated GUI testing is widely used to help ensure the quality of mobile apps. However, many GUIs require appropriate text inputs to proceed to the next page, which remains a prominent obstacle for testing coverage. Considering the diversity and semantic requirement of valid inputs (e.g., flight departure, movie name), it is challenging to automate the text input generation. Inspired by the fact that the pre-trained Large Language Model (LLM) has made outstanding progress in text generation, we propose an approach named QTypist based on LLM for intelligently generating semantic input text according to the GUI context. To boost the performance of LLM in the mobile testing scenario, we develop a prompt-based data construction and tuning method which automatically extracts the prompts and answers for model tuning. We evaluate QTypist on 106 apps from Google Play, and the result shows that the passing rate of QTypist is 87%, which is 93% higher than the best baseline. We also integrate QTypist with the automated GUI testing tools and it can cover 42% more app activities, 52% more pages, and subsequently help reveal 122% more bugs compared with the raw tool.

Mobile apps are indispensable for people's daily life. With the increase of GUI functions, apps have become more complex and diverse. As the Android app is event-driven, Activity Transition Graph (ATG) becomes an important way of app abstract and graphical user interface (GUI) modeling. Although existing works provide static and dynamic analysis to build ATG for applications, the completeness of ATG obtained is poor due to the low coverage of these techniques. To tackle this challenge, we propose a novel approach, ArchiDroid, to automatically augment the ATG via graph convolution network. It models both the semantics of activities and the graph structure of activity transitions to predict the transition between activities based on the seed ATG extracted by static analysis. The evaluation demonstrates that ArchiDroid can achieve 86% precision and 94% recall in predicting the transition between activities for augmenting ATG. We further apply the augmented ATG in two downstream tasks, i.e., guidance in automated GUI testing and assistance in app function design. Results show that the automated GUI testing tool integrated with ArchiDroid achieves 43% more activity coverage and detects 208% more bugs. Besides, ArchiDroid can predict the missing transition with 85% accuracy in real-world apps for assisting the app function design, and an interview case study further demonstrates its usefulness.

Existing automatic code comment generators mainly focus on producing a general description of functionality for a given code snippet without considering developer intentions. However, in real-world practice, comments are complicated, which often contain information reflecting various intentions of developers, e.g., functionality summarization, design rationale, implementation details, code properties, etc. To bridge the gap between automatic code comment generation and real-world comment practice, we define Developer-Intent Driven Code Comment Generation, which can generate intent-aware comments for the same source code with different intents. To tackle this challenging task, we propose DOME, an approach that utilizes Intent-guided Selective Attention to explicitly select intent-relevant information from the source code, and produces various comments reflecting different intents. Our approach is evaluated on two real-world Java datasets, and the experimental results show that our approach outperforms the state-of-the-art baselines. A human evaluation also confirms the significant potential of applying DOME in practical usage, enabling developers to comment code effectively according to their own needs.

Nowadays increasing AI techniques, e.g., reinforcement learning, imitation learning, etc., are applied to solve sequential decision-making problems by modeling them as Markov Decision Process (MDP), and achieve superior performance in areas, such as video games, robotics and autonomous driving etc. The reliability of such models is facing severe challenges especially in some safety-critical areas, where failures would bring intolerable disasters. Existing works testing episodic decision-making models are not workable, since they neglect the nature of sequentiality and interactivity in MDP. While other works testing sequential decision-making models are challenged by low testing efficiency because the interaction of MDP is time-consuming. In this paper, we propose an optimized fuzzing framework SeqDivFuzz which infers the sequence diversity during the MDP interaction process to effectively and efficiently test sequential decision-making models in blackbox settings. It adapts the existing fuzzing framework, including Seed Selection, Seed Mutation, Feedback Analysis and integrating a module of Diversity Inference to accelerate the fuzzing procedure. The module learns historical in-process information to check the diversity of test cases when running up to checkPoint in the course of MDP, and early terminating those non-diverse ones. We conduct experimental evaluation with four models involving three simulation environments. The results reflect that SeqDivFuzz exposes 12.3%~49.1% more crashes during a 12-hour testing procedure in four pairs of models and environments compared with the state-of-the-art fuzzing framework. The idea of in-process terminating can potentially boost other techniques for testing sequential decision-making models.

Secure coding practices (SCPs) have been proposed to guide software developers to write code securely to prevent potential security vulnerabilities. Yet, they are typically one-sentence principles without detailed specifications, e.g., “Properly free allocated memory upon the completion of functions and at all exit points.”, which makes them difficult to follow in practice, especially for software developers who are not yet experienced in secure programming. To address this problem, this paper proposes SCPatcher, an automated approach to enrich secure coding practices by mining crowd security discussions on online knowledge-sharing platforms, such as Stack Overflow. In particular, for each security post, SCPatcher first extracts the area of coding examples and coding explanations with a fix-prompt tuned Large Language Model (LLM) via Prompt Learning. Then, it hierarchically slices the lengthy code into coding examples and summarizes the coding explanations with the areas. Finally, SCPatcher matches the CWE and Public SCP, integrating them with extracted coding examples and explanations to form the SCP specifications, which are the wild SCPs with details, proposed by the developers. To evaluate the performance of SCPatcher, we conduct experiments on 3,907 security posts from Stack Overflow. The experimental results show that SCPatcher outperforms all baselines in extracting the coding examples with 2.73 % MLine on average, as well as coding explanations with 3.97 % F1 on average. Moreover, we apply SCPatcher on 447 new security posts to further evaluate its practicality, and the extracted SCP specifications enrich the public SCPs with 3,074 lines of code and 1,967 sentences.

In open source software (OSS) communities, existing leadership indicators are dominantly measured by code contribution or community influence. Recent studies on emergent leadership shed light on additional dimensions such as intellectual stimulation in collaborative communications. This paper aims to mine communication styles and identify emergent leadership behaviors in OSS communities, using issue comments data. We start with the construction of 6 categories of leadership behaviors based on existing leadership studies. Then, we manually label leadership behaviors in 10,000 issue comments from 10 OSS projects, and extract 304 heuristic linguistic patterns which represent different types of emergent leadership behaviors in flexible and concise manners. Next, an automated algorithm is developed to merge and consolidate different pattern sets extracted from multiple projects into a final pattern ranking list, which can be applied for the automatic leadership identification. The evaluation results show that iLead can achieve a median precision of 0.82 and recall of 0.78, outperforming ten machine/deep learning baselines. We argue that emergent leadership behaviors in issue discussion should be taken into consideration to broaden existing OSS leadership viewpoints.

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the upgrading of mobile devices and the development of aesthetics, the visual effects of the GUI are more and more attracting, and users pay more attention to the accessibility and usability of applications. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, component occlusion, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a fully automated approach,Nighthawk, based on deep learning for modelling visual information of the GUI screenshot.Nighthawkcan detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. At the same time, training the model needs a large amount of labeled buggy screenshots, which requires considerable manual effort to prepare them. We therefore propose a heuristic-based training data auto-generation method to automatically generate the labeled training data. The evaluation demonstrates that ourNighthawkcan achieve average 0.84 precision and 0.84 recall in detecting UI display issues, average 0.59 AP and 0.60 AR in localizing these issues. We also evaluateNighthawkwith popular Android apps on Google Play and F-Droid, and successfully uncover 151 previously-undetected UI display issues with 75 of them being confirmed or fixed so far.

Mobile apps are indispensable for people’s daily life. Complementing with automated GUI testing, manual testing is the last line of defence for app quality. However, the repeated actions and easily missing of functionalities make manual testing time-consuming and inefficient. Inspired by the game candy crush with flashy candies as hint moves for players, we propose an approach named NaviDroid for navigating testers via highlighted next operations for more effective and efficient testing. Within NaviDroid, we construct an enriched state transition graph with the triggering actions as the edges for two involved states. Based on it, we utilize the dynamic programming algorithm to plan the exploration path, and augment the GUI with visualized hints for testers to quickly explore untested activities and avoid duplicate explorations. The automated experiments demonstrate the high coverage and efficient path planning of NaviDroid and a user study further confirms its usefulness. The NaviDroid can help us develop more robust software that works in more mission-critical settings, not only by performing more thorough testing with the same effort that has been put in before, but also by integrating these techniques into different parts of development pipeline.

Background. Issues are prevalent, and identifying the correct priority of the reported issues is crucial to reduce the maintenance effort and ensure higher software quality. There are several approaches for the automatic priority prediction, yet they do not fully utilize the related information that might influence the priority assignment. Our observation reveals that there are noticeable correlations between an issue’s priority and its category, e.g., an issue of bug category tends to be assigned with higher priority than an issue of document category. This correlation motivates us to employ multi-task learning to share the knowledge about issue’s category prediction and facilitating priority prediction.

In community-based software development, developers frequently rely on live-chatting to discuss emergent bugs/errors they encounter in daily development tasks. However, it remains a challenging task to accurately record such knowledge due to the noisy nature of interleaved dialogs in live chat data. In this paper, we first formulate the task of identifying and synthesizing bug reports from community live chats, and propose a novel approach, named BugListener, to address the challenges. Specifically, BugListener automates three sub-tasks: 1) Disentangle the dialogs from massive chat logs by using a Feed-Forward neural network; 2) Identify the bug-report dialogs from separated dialogs by leveraging the Graph neural network to learn the contextual information; 3) Synthesize the bug reports by utilizing Transfer Learning techniques to classify the sentences into: observed behaviors (OB), expected behaviors (EB), and steps to reproduce the bug (SR). BugListener is evaluated on six open source projects. The results show that: for bug report identification, BugListener achieves the average F1 of 77.74%, improving the best baseline by 12.96%; and for bug report synthesis task, BugListener could classify the OB, EB, and SR sentences with the F1 of 84.62%, 71.46%, and 73.13%, improving the best baselines by 9.32%, 12.21%, 10.91%, respectively. A human evaluation study also confirms the effectiveness of BugListener in generating relevant and accurate bug reports. These demonstrate the significant potential of applying BugListener in community-based software development, for promoting bug discovery and quality improvement.

User reviews of mobile apps provide a communication channel for developers to perceive user satisfaction. Many app features that users have problems with are usually expressed by key phrases such as "upload pictures", which could be buried in the review texts. The lack of fine-grained view about problematic features could obscure the developers' understanding of where the app is frustrating users, and postpone the improvement of the apps. Existing pattern-based approaches to extract target phrases suffer from low accuracy due to insufficient semantic understanding of the reviews, thus can only summarize the high-level topics/aspects of the reviews. This paper proposes a semantic-aware, fine-grained app review analysis approach (SIRA) to extract, cluster, and visualize the problematic features of apps. The main component of SIRA is a novel BERT+Attr-CRF model for fine-grained problematic feature extraction, which combines textual descriptions and review attributes to better model the semantics of reviews and boost the performance of the traditional BERT-CRF model. SIRA also clusters the extracted phrases based on their semantic relations and presents a visualization of the summaries. Our evaluation on 3,426 reviews from six apps confirms the effectiveness of SIRA in problematic feature extraction and clustering. We further conduct an empirical study with SIRA on 318,534 reviews of 18 popular apps to explore its potential application and examine its usefulness in real-world practice.

The main goal of dialogue disentanglement is to separate the mixed utterances from a chat slice into independent dialogues. Existing models often utilize either an utterance-to-utterance (U2U) prediction to determine whether two utterances that have the “reply-to” relationship belong to one dialogue, or an utterance-to-thread (U2T) prediction to determine which dialogue-thread a given utterance should belong to. Inspired by mutual leaning, we propose MuiDial, a novel dialogue disentanglement model, to exploit the intent of each utterance and feed the intent to a mutual learning U2U-U2T disentanglement model. Experimental results and in-depth analysis on several benchmark datasets demonstrate the effectiveness and generalizability of our approach.

Static bug finders (also known as static code analyzers, e.g., Find-Bugs, SonarQube) have been widely-adopted by developers to find bugs in real-world software projects. They leverage predefined heuristic static analysis rules to scan source code or binary code of a software project, and report violations to these rules as warnings to be verified. However, the advantages of static bug finders are overshadowed by such issues as uncovered obvious bugs, false positives, etc. To improve these tools, many techniques have been proposed to filter out false positives reported or design new static analysis rules. Nevertheless, the under-performance of bug finders can also be caused by the incorrectness of current rules contained in the static bug finders, which is not explored yet. In this work, we propose a differential testing approach to detect bugs in the rules of four widely-used static bug finders, i.e., SonarQube, PMD, SpotBugs, and ErrorProne, and conduct a qualitative study about the bugs found. The experiment on 2,728 open source projects reveals 46 bugs in the static bug finders, among which 30 are fixed or confirmed and the left are awaiting confirmation. We also summarize 13 bug patterns in the static analysis rules based on their context and root causes, which can serve as the checklist for designing and implementing other rules and/or in other tools. This study indicates that the commonly-used static bug finders are not as reliable as they might have been envisaged. It not only demonstrates the effectiveness of our approach, but also highlights the need to continue improving the reliability of the static bug finders.

Deliberation is a common and natural behavior in human daily life. For example, when writing papers or articles, we usually first write drafts, and then iteratively polish them until satisfied. In light of such a human cognitive process, we propose DECOM, which is a multi-pass deliberation framework for automatic comment generation. DECOM consists of multiple Deliberation Models and one Evaluation Model. Given a code snippet, we first extract keywords from the code and retrieve a similar code fragment from a pre-defined corpus. Then, we treat the comment of the retrieved code as the initial draft and input it with the code and keywords into DECOM to start the iterative deliberation process. At each deliberation, the deliberation model polishes the draft and generates a new comment. The evaluation model measures the quality of the newly generated comment to determine whether to end the iterative process or not. When the iterative process is terminated, the best-generated comment will be selected as the target comment. Our approach is evaluated on two real-world datasets in Java (87K) and Python (108K), and experiment results show that our approach outperforms the state-of-the-art baselines. A human evaluation study also confirms the comments generated by DECOM tend to be more readable, informative, and useful.

As the bridge between users and software, Graphical User Interface (GUI) is critical to the app accessibility. Scaling up the font or display size of GUI can help improve the visual impact, readability, and usability of an app, and is frequently used by the elderly and people with vision impairment. Yet this can easily lead to scaling issues such as text truncation, component overlap, which negatively influence the acquirement of the right information and the fluent usage of the app. Previous techniques for UI display issue detection and cross-platform inconsistency detection cannot work well for these scaling issues. In this paper, we propose an automated method, dVermin, for scaling issue detection, through detecting the inconsistency of a view under the default and a larger display scale. The evaluation result shows that dVermin achieves 97% precision and 97% recall in issue page detection, and 84% precision and 91% recall for issue view detection, outperforming two state-of-the-art baselines by a large margin. We also evaluate dVermin with popular Android apps on F-droid, and successfully uncover 21 previously-undetected scaling issues with 20 of them being confirmed/fixed.

Code summarization, the task of generating useful comments given the code, has long been of interest. Most of the existing code summarization models are trained and validated on widely-used code comment benchmark datasets. However, little is known about the quality of the benchmark datasets built from real-world projects. Are the benchmark datasets as good as expected? To bridge the gap, we conduct a systematic research to assess and improve the quality of four benchmark datasets widely used for code summarization tasks. First, we propose an automated code-comment cleaning tool that can accurately detect noisy data caused by inappropriate data preprocessing operations from existing benchmark datasets. Then, we apply the tool to further assess the data quality of the four benchmark datasets, based on the detected noises. Finally, we conduct comparative experiments to investigate the impact of noisy data on the performance of code summarization models. The results show that these data preprocessing noises widely exist in all four benchmark datasets, and removing these noisy data leads to a significant improvement on the performance of code summarization. We believe that the findings and insights will enable a better understanding of data quality in code summarization tasks, and pave the way for relevant research and practice.

Natural language (NL) documentation is the bridge between software managers and testers, and NL test cases are prevalent in system-level testing and other quality assurance activities. Due to reasons such as requirements redundancy, parallel testing, tester turn-over within long evolving history, there are inevitably lots of redundant test cases, which significantly increase the cost. Previous redundancy detection approaches typically treat the textual descriptions as a whole to compare their similarity and suffer from low precision. Our observation reveals that a test case can have explicit test-oriented entities, such as tested function Components, Constraints, etc; and there are also specific relations between these entities. This inspires us with a potential opportunity for accurate redundancy detection. In this paper, we first define five test-oriented entity categories and four associated relation categories, and re-formulate the NL test case redundancy detection problem as the comparison of detailed testing content guided by the test-oriented entities and relations. Following that, we propose Tscope, a fine-grained approach for redundant NL test case detection by dissecting test cases into atomic test tuple(s) with the entities restricted by associated relations. To serve as the test case dissection, Tscope designs a context-aware model for the automatic entity and relation extraction. Evaluation on 3,467 test cases from ten projects shows Tscope could achieve 91.8% precision, 74.8% recall and 82.4% F1, significantly outperforming state-of-the-art approaches and commonly-used classifiers. This new formulation of the NL test case redundant detection problem can motivate the follow-up studies in further improving this task and other related tasks involving NL descriptions.

Identifying and optimizing open participation is essential to the success of open software development. Existing studies highlighted the importance of worker recommendation for crowdtesting tasks in order to improve bug detection efficiency, i.e., detect more bugs with fewer workers. However, there are a couple of limitations in existing work. First, these studies mainly focus on one-time recommendations based on expertise matching at the beginning of a new task. Second, the recommendation results suffer from severe popularity bias, i.e., highly experienced workers are recommended in almost all the tasks, while less experienced workers rarely get recommended. This article argues the need for context- and fairness-aware in-process crowdworker recommendation in order to address these limitations. We motivate this study through a pilot study, revealing the prevalence of long-sized non-yielding windows, i.e., no new bugs are revealed in consecutive test reports during the process of a crowdtesting task. This indicates the potential opportunity for accelerating crowdtesting by recommending appropriate workers in a dynamic manner, so that the non-yielding windows could be shortened. Besides, motivated by the popularity bias in existing crowdworker recommendation approach, this study also aims at alleviating the unfairness in recommendations. Driven by these observations, this article proposes a context- and fairness-aware in-process crowdworker recommendation approach, iRec2.0, to detect more bugs earlier, shorten the non-yielding windows, and alleviate the unfairness in recommendations. It consists of three main components: (1) the modeling of dynamic testing context, (2) the learning-based ranking component, and (3) the multi-objective optimization-based re-ranking component. The evaluation is conducted on 636 crowdtesting tasks from one of the largest crowdtesting platforms, and results show the potential of iRec2.0 in improving the cost-effectiveness of crowdtesting by saving the cost, shortening the testing process, and alleviating the unfairness among workers. In detail, iRec2.0 could shorten the non-yielding window by a median of 50%–66% in different application scenarios, and consequently have potential of saving testing cost by a median of 8%–12%. Meanwhile, the recommendation frequency of the crowdworker drop from 34%–60% to 5%–26% under different scenarios, indicating its potential in alleviating the unfairness among crowdworkers.

Crowdsourced software testing (short for crowdtesting) is a special type of crowdsourcing. It requires that crowdworkers master appropriate skill-sets and commit significant effort for completing a task. Abundant uncertainty may arise during a crowdtesting process due to imperfect information between the task requester and crowdworkers. For example, a worker frequently chooses tasks in an ad hoc manner in crowdtesting context, and an inappropriate task selection may lead to the worker's failing to detect any bugs, and significant testing effort unpaid and wasted. Recent studies have explored methods for supporting task requesters to make informed decisions on task pricing, worker recommendation, and so on. Unfortunately, very few study offers decision making support from the crowdworkers’ perspectives. We motivate this study through a pilot study, revealing the large portion (74 percent) of unpaid crowdworkers’ effort due to the inappropriate task choice. Drawn from our previous work on context-aware crowdworker recommendations, we advocate a more effective alternative to manual task selection would be to provide contextualized and personalized task recommendation considering the diverse distribution of worker preference and expertise, with objectives to increase their winning chances and to potentially reduce the frequency of unpaid crowd work. This paper proposes a context-aware personalized task recommendation approachPTRec, consisting of a testing context model and a learning-based task recommendation model to aid dynamic worker decision in selecting crowdtesting tasks. The testing context model is constructed in two perspectives, i.e., process context and resource context, to capture the in-process progress-oriented information and crowdworkers’ characteristics respectively. Built on top of this context model, the learning-based task recommendation model extracts 60 features automatically, and employs random forest learner to generate dynamic and personalized task recommendation which matches workers’ expertise and interest. The evaluation is conducted on 636 crowdtesting tasks involving 2,404 crowdworkers from one of the largest crowdtesting platforms, and results show our approach can achieve an average precision of 82 percent, average recall of 84 percent, and save an estimated average of 81 percent effort originally spent on exploring, significantly outperforming four commonly-used and state-of-the-art baselines. This indicates its potential in recommending proper tasks to workers so as to improve bug detection efficiency and increase their monetary earnings.

Traceability links between requirements and source code are beneficial to the maintenance and evolution activities. Compared with the proposed unsupervised solutions, supervised solutions are more effective to generate trace links automatically and gaining more attention. However, supervised solutions often need to spend a lot of effort on labeling data. To overcome this limitation, we propose a self-enhanced automatic traceability link recovery approach based on structure knowledge mining for small-scale labeled data, named K2Trace, which not only enhances the semantic representations of artifacts by mining context information but also self-enhances the size of training set by exploring transitive relationships. Evaluation results show that K2Trace can outperform the state-of-the-art baseline approach. K2Trace proves the usefulness of mining knowledge from the structure information of software artifacts, as well as provides a new way to substantially reduce the amount of training data needed for training efficient classification models, which may pave the way for generating accurate trace links.

Background. Where to start contributing to a project is a critical challenge for newcomers of open source projects. To support newcomers, GitHub utilizes the Good First Issue (GFI) label, with which project members can manually tag issues in an open source project that are suitable for the newcomers. However, manually labeling GFIs is time- and effort-consuming given the large number of candidate issues. In addition, project members need to have a close understanding of the project to label GFIs accurately.

Despite the valuable information contained in software chat messages, disentangling them into distinct conversations is an essential prerequisite for any in-depth analyses that utilize this information. To provide a better understanding of the current state-of-the-art, we evaluate five popular dialog disentanglement approaches on software-related chat. We find that existing approaches do not perform well on disentangling software-related dialogs that discuss technical and complex topics. Further investigation on how well the existing disentanglement measures reflect human satisfaction shows that existing measures cannot correctly indicate human satisfaction on disentanglement results. Therefore, in this paper, we introduce and evaluate a novel measure, named DLD. Using results of human satisfaction, we further summarize four most frequently appeared bad disentanglement cases on software-related chat to insight future improvements. These cases include (i) Ignoring Interaction Patterns, (ii) Ignoring Contextual Information, (iii) Mixing up Topics, and (iv) Ignoring User Relationships. We believe that our findings provide valuable insights on the effectiveness of existing dialog disentanglement approaches and these findings would promote a better application of dialog disentanglement in software engineering.

Collaborative live chats are gaining popularity as a development communication tool. In community live chatting, developers are likely to post issues they encountered (e.g., setup issues and compile issues), and other developers respond with possible solutions. Therefore, community live chats contain rich sets of information for reported issues and their corresponding solutions, which can be quite useful for knowledge sharing and future reuse if extracted and restored in time. However, it remains challenging to accurately mine such knowledge due to the noisy nature of interleaved dialogs in live chat data. In this paper, we first formulate the problem of issue-solution pair extraction from developer live chat data, and propose an automated approach, named ISPY, based on natural language processing and deep learning techniques with customized enhancements, to address the problem. Specifically, ISPY automates three tasks: 1) Disentangle live chat logs, employing a feedforward neural network to disentangle a conversation history into separate dialogs automatically; 2) Detect dialogs discussing issues, using a novel convolutional neural network (CNN), which consists of a BERT-based utterance embedding layer, a context-aware dialog embedding layer, and an output layer; 3) Extract appropriate utterances and combine them as corresponding solutions, based on the same CNN structure but with different feeding inputs. To evaluate ISPY, we compare it with six baselines, utilizing a dataset with 750 dialogs including 171 issue-solution pairs and evaluate ISPY from eight open source communities. The results show that, for issue-detection, our approach achieves the F1 of 76%, and outperforms all baselines by 30%. Our approach achieves the F1 of 63% for solution-extraction and outperforms the baselines by 20%. Furthermore, we apply ISPY on three new communities to extensively evaluate ISPY’s practical usage. Moreover, we publish over 30K issue-solution pairs extracted from 11 communities. We believe that ISPY can facilitate community-based software development by promoting knowledge sharing and shortening the issue-resolving process.

Background: When submitting patches for code review, a patch is expected as the patch that is accepted in the fewest reworks (rounds of revisions) possibly. However, in practice, not all submitted patches can be accepted and integrated into the codebase the first time, since their quality is unsatisfactory or even poor, which might not pass patch review or need multiple revisions. Aims: In order to improve the effectiveness and efficiency of modern code review, this paper aims to identify the factors that impact the the patch acceptance and the rounds of revisions. Method: We first extract 29 factors from four categories: author, patch, static code quality and review. Then, we apply statistical approach to examine which of and how these factors impact the patch acceptance and the rounds of revisions, respectively. Results: Through case studies of 16,950 patches of 18 projects from Android and Eclipse systems, the result shows that: 1) the number of accepted/reviewed patches by an author has a statistically significant effect on the patch acceptance; 2) the kinds of languages and the number of invited reviewers have a statistically significant effect on the rounds of the revisions; 3) static code quality of patches has little impact on both patch acceptance and the rounds of revisions, although complexity has impact on the number of invited reviewers. Besides, there are other factors that have impact on the patch acceptance and rounds of revisions for different systems. Conclusions: Based on the analysis result, this study offers some suggestions for researchers and practitioners to improve the quality of submitted patches and the process of code review.

Modern communication platforms such as Gitter and Slack play an increasingly critical role in supporting software teamwork, especially in open source development.Conversations on such platforms often contain intensive, valuable information that may be used for better understanding OSS developer communication and collaboration. However, little work has been done in this regard. To bridge the gap, this paper reports a first comprehensive empirical study on developers' live chat, investigating when they interact, what community structures look like, which topics are discussed, and how they interact. We manually analyze 749 dialogs in the first phase, followed by an automated analysis of over 173K dialogs in the second phase. We find that developers tend to converse more often on weekdays, especially on Wednesdays and Thursdays (UTC), that there are three common community structures observed, that developers tend to discuss topics such as API usages and errors, and that six dialog interaction patterns are identified in the live chat communities. Based on the findings, we provide recommendations for individual developers and OSS communities, highlight desired features for platform vendors, and shed light on future research directions. We believe that the findings and insights will enable a better understanding of developers' live chat, pave the way for other researchers, as well as a better utilization and mining of knowledge embedded in the massive chat history.

Graphical User Interface (GUI) provides visual bridges between software apps and end users. However, due to the compatibility of software or hardware, UI display issues such as text overlap, blurred screen, image missing always occur during GUI rendering on different devices. Because these UI display issues can be found directly by human eyes, in this paper, we implement an online UI display issue detection tool OwlEyes-Online, which provides a simple and easy-to-use platform for users to realize the automatic detection and localization of UI display issues. The OwlEyes-Online can automatically run the app and get its screenshots and XML files, and then detect the existence of issues by analyzing the screenshots. In addition, OwlEyes-Online can also find the detailed area of the issue in the given screenshots to further remind developers. Finally, OwlEyes-Online will automatically generate test reports with UI display issues detected in app screenshots and send them to users. The OwlEyes-Online was evaluated and proved to be able to accurately detect UI display issues. Tool Link: http://www.owleyes.online:7476 Github Link: https://github.com/franklinbill/owleyes Demo Video Link: https://youtu.be/002nHZBxtCY

Crowdsourced testing is an emerging trend, in which test tasks are entrusted to the online crowd workers. Typically, a crowdsourced test task aims to detect as many bugs as possible within a limited budget. However not all crowd workers are equally skilled at finding bugs; Inappropriate workers may miss bugs, or report duplicate bugs, while hiring them requires nontrivial budget. Therefore, it is of great value to recommend a set of appropriate crowd workers for a test task so that more software bugs can be detected with fewer workers. This paper first presents a new characterization of crowd workers and characterizes them with testing context, capability, and domain knowledge. Based on the characterization, we then propose Multi-Objective Crowd wOrker recoMmendation approach (MOCOM), which aims at recommending a minimum number of crowd workers who could detect the maximum number of bugs for a crowdsourced testing task. Specifically, MOCOM recommends crowd workers by maximizing the bug detection probability of workers, the relevance with the test task, the diversity of workers, and minimizing the test cost. We experimentally evaluate MOCOM on 532 test tasks, and results show that MOCOM significantly outperforms five commonly-used and state-of-the-art baselines. Furthermore, MOCOM can reduce duplicate reports and recommend workers with high relevance and larger bug detection probability; because of this it can find more bugs with fewer workers.

Background: Given the invisibility and unpredictability of distributed crowdtesting processes, there is a large number of duplicate reports, and detecting these duplicate reports is an important task to help save testing effort. Although, many approaches have been proposed to automatically detect the duplicates, the comparison among them and the practical guidelines to adopt these approaches in crowdtesting remain vague.

Online chatting is gaining popularity and plays an increasingly significant role in software development. When discussing functionalities, developers might reveal their desired features to other developers. Automated mining techniques towards retrieving feature requests from massive chat messages can benefit the requirements gathering process. But it is quite challenging to perform such techniques because detecting feature requests from dialogues requires a thorough understanding of the contextual information, and it is also extremely expensive on annotating feature-request dialogues for learning. To bridge that gap, we recast the traditional text classification task of mapping single dialog to its class into the task of determining whether two dialogues are similar or not by incorporating few-shot learning. We propose a novel approach, named FRMiner, which can detect feature-request dialogues from chat messages via deep Siamese network. We design a BiLSTM-based dialog model that can learn the contextual information of a dialog in both forward and reverse directions. Evaluation on the real-world projects shows that our approach achieves average precision, recall and F1-score of 88.52%, 88.50% and 88.51%, which confirms that our approach could effectively detect hidden feature requests from chat messages, thus can facilitate gathering comprehensive requirements from the crowd in an automated way.

Identifying and optimizing open participation is essential to the success of open software development. Existing studies highlighted the importance of worker recommendation for crowdtesting tasks in order to detect more bugs with fewer workers. However, these studies mainly focus on one-time recommendations with respect to the initial context at the beginning of a new task. This paper argues the need for in-process crowdtesting worker recommendation. We motivate this study through a pilot study, revealing the prevalence of long-sized non-yielding windows, i.e., no new bugs are revealed in consecutive test reports during the process of a crowdtesting task. This indicates the potential opportunity for accelerating crowdtesting by recommending appropriate workers in a dynamic manner, so that the non-yielding windows could be shortened.

Requirement entities, "explicit specification of concepts that define the primary function objects", play an important role in requirement analysis for software development and maintenance. It is a labor-intensive activity to extract requirement entities from textual requirements, which is typically done manually. A few existing studies propose automated methods to support key requirement concept extraction. However, they face two main challenges: lack of domain-specific natural language processing techniques and expensive labeling effort. To address the challenges, this study presents a novel approach named RENE, which employs LSTM-CRF model for requirement entity extraction and introduces the general knowledge to reduce the demands for labeled data. It consists of four phases: 1) Model construction, where RENE builds LSTM-CRF model and an isomorphic LSTM language model for transfer learning; 2) LSTM language model training, where RENE captures general knowledge and adapt to requirement context; 3) LSTM-CRF training, where RENE trains the LSTM-CRF model with the transferred layers; 4) Requirement entity extraction, where RENE applies the trained LSTM-CRF model to a new-coming requirement, and automatically extracts its requirement entities. RENE is evaluated using two methods: evaluation on historical dataset and user study. The evaluation on the historical dataset shows that RENE could achieve 79% precision, 81% recall, and 80% F1. The evaluation results from the user study also suggest that RENE could produce more accurate and comprehensive requirement entities, compared with those produced by engineers.

Pull requests (PRs) selection is a challenging task faced by integrators in pull-based development (PbD), with hundreds of PRs submitted on a daily basis to large open-source projects. Managing these PRs manually consumes integrators' time and resources and may lead to delays in the acceptance, response, or rejection of PRs that can propose bug fixes or feature enhancements. On the one hand, well-known platforms for performing PbD, like GitHub, do not provide built-in recommendation mechanisms for facilitating the management of PRs. On the other hand, prior research on PRs recommendation has focused on the likelihood of either a PR being accepted or receive a response by the integrator. In this paper, we consider both those likelihoods, this to help integrators in the PRs selection process by suggesting to them the appropriate actions to undertake on each specific PR. To this aim, we propose an approach, called CARTESIAN (aCceptance And Response classificaTion-based requESt IdentificAtioN) modeling the PRs recommendation according to PR actions. In particular, CARTESIAN is able to recommend three types of PR actions: accept, respond, and reject. We evaluated CARTESIAN on the PRs of 19 popular GitHub projects. The results of our study demonstrate that our approach can identify PR actions with an average precision and recall of about 86%. Moreover, our findings also highlight that CARTESIAN outperforms the results of two baseline approaches in the task of PRs selection.

The ability in rapidly learning and adapting to evolving user needs is key to modern business successes. Existing methods are based on text mining and machine learning techniques to analyze user comments and feedback, and often constrained by heavy reliance on manually codified rules or insufficient training data. Multitask learning (MTL) is an effective approach with many successful applications, with the potential to address these limitations associated with requirements analysis tasks. In this paper, we propose a deep MTL-based approach, DEMAR, to address these limitations when discovering requirements from massive issue reports and annotating the sentences in support of automated requirements analysis. DEMAR consists of three main phases: (1) data augmentation phase, for data preparation and allowing data sharing beyond single task learning; (2) model construction phase, for constructing the MTL-based model for requirements discovery and requirements annotation tasks; and (3) model training phase, enabling eavesdropping by shared loss function between the two related tasks. Evaluation results from eight open-source projects show that, the proposed multitask learning approach outperforms two state-of-the-art approaches (CNC and FRA) and six common machine learning algorithms, with the precision of 91% and the recall of 83% for requirements discovery task, and the overall accuracy of 83% for requirements annotation task. The proposed approach provides a novel and effective way to jointly learn two related requirements analysis tasks. We believe that it also sheds light on further directions of exploring multitask learning in solving other software engineering problems.

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our OwlEye. The evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate OwlEye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.

Pull requests (PRs) prioritization is one of the main challenges faced by integrators in pull-based development. This is especially true for large open-source projects where hundreds of pull requests are submitted daily. Indeed, managing these pull requests manually consumes time and resources and may lead to delays in the reaction (i.e., acceptance or response) to enhancements or bug fixes suggested in the codebase by contributors. We propose an approach, called AR-Prioritizer (Acceptance and Response based Prioritizer), integrating a PRs prioritization mechanism that considers these two aspects. The results of our study demonstrate that our approach can recommend top@5, top@10, and top@20 most likely to be accepted and responded pull requests with Mean Average Precision of 95.3%, 89.6%, and 79.6% and Average Recall of 40%, 65.7%, and 92.9%. Moreover, AR-Prioritizer has outperformed the baseline models with a statistical significance in prioritizing the most likely to be accepted and responded to PRs.

Peer code review has been proven to be an effective practice for quality assurance, and widely adopted by commercial companies and open source communities as GitHub. However, identifying an appropriate code reviewer for a pull request is a non-trivial task considering the large number of candidate reviewers. Several approaches have been proposed for reviewer recommendation, yet none of them has conducted a complete comparison to explore which one is more effective. This paper aims at conducting an experimental evaluation of the commonly-used and state-of-the-art approaches for code reviewer recommendation. We begin with a systematic review of approaches for code reviewer recommendation, and choose six approaches for experimental evaluation. We then implement these approaches and conduct reviewer recommendation on 12 large-scale open source projects with 53,005 pull requests spanning two years. Results show that there is no golden rule when selecting code reviewer recommendation approaches, and the best approach varies in terms of different evaluation metrics (e.g., Top-5 Accuracy, MRR) and experimental projects. Nevertheless, TIE, which utilizes the textual similarity and file path similarity, is the most promising one. We also explore the sensitivity of these approaches to training data, and compare their time cost. This approach provides new insights and practical guidelines for choosing approaches for reviewer recommendation.

Requirements are usually written by different stakeholders with diverse backgrounds and skills and evolve continuously. Therefore inconsistency caused by specialized jargons and different domains, is inevitable. In particular, entity coreference in Requirement Engineering (RE) is that different linguistic expressions refer to the same real-world entity. It leads to misconception about technical terminologies, and impacts the readability and understandability of requirements negatively. Manual detection entity coreference is labor-intensive and time-consuming. In this paper, we propose a DEEP context-wise semantic method named DeepCoref to entity COREFerence detection. It consists of one fine-tuning BERT model for context representation and a Word2Vec-based network for entity representation. We use a multi-layer perception in the end to fuse and make a trade-off between two representations for obtaining a better representation of entities. The input of the network is requirement contextual text and related entities, and the output is the predictive label to infer whether two entities are coreferent. The evaluation on industry data shows that our approach significantly outperforms three baselines with average precision and recall of 96.10% and 96.06% respectively. We also compare DeepCoref with three variants to demonstrate the performance enhancement from different components.

In practice, it is very important to determine the size of a proposed software system yet to be built based on its requirements, i.e., early in the development life cycle. The most widely used approach for size estimation is Function Point Analysis (FPA). However, since FPA involves human judgment, the estimation results are some degree of subjective, and the process is labor and cost intensive. In this paper, we propose a novel approach to identify transaction functions from textual requirements automatically by leveraging a set of natural language processing techniques and machine learning models. We evaluate our approach on 1,864 requirements and 104,691 transaction functions taken from 36 financial projects from one banking industry. The results show that the contents of the suggested transaction functions by our approach are high in quality, with low perplexity value of 8.5 and high BLEU score of 34 on average. The types of suggested transaction functions can also be accurately classified, with overall accuracy of 0.99 on average. Our approach can provide reasonable suggestions that assist industrial practitioners to identify transaction functions faster and easier.

Software engineers get questions of “how much testing is enough” on a regular basis. Existing approaches in software testing management employ experience-, risk-, or value-based analysis to prioritize and manage testing processes. However, very few is applicable to the emerging crowdtesting paradigm to cope with extremely limited information and control over unknown, online crowdworkers. In practice, deciding when to close a crowdtesting task is largely done by experience-based guesswork and frequently results in ineffective crowdtesting. More specifically, it is found that an average of 32% testing cost was wasteful spending in current crowdtesting practice. This article intends to address this challenge by introducing automated decision support for monitoring and determining appropriate time to close crowdtesting tasks. To that end, it first investigates the necessity and feasibility of close prediction of crowdtesting tasks based on an industrial dataset. Next, it proposes a close prediction approach named iSENSE2.0, which applies incremental sampling technique to process crowdtesting reports arriving in chronological order and organizes them into fixed-sized groups as dynamic inputs. Then, a duplicate tagger analyzes the duplicate status of received crowd reports, and a CRC-based (Capture-ReCapture) close estimator generates the close decision based on the dynamic bug arrival status. In addition, a coverage-based sanity checker is designed to reinforce the stability and performance of close prediction. Finally, the evaluation of iSENSE2.0 is conducted on 56,920 reports of 306 crowdtesting tasks from one of the largest crowdtesting platforms. The results show that a median of 100% bugs can be detected with 30% saved cost. The performance of iSENSE2.0 does not demonstrate significant difference with the state-of-the-art approach iSENSE , while the later one relies on the duplicate tag, which is generally considered as time-consuming and tedious to obtain.

Requirements traceability provides important support throughout all software life cycle; however, creating such links manually is time-consuming and error-prone. Supervised automated solutions use machine learning or deep learning techniques to generate trace links, but require large labeled dataset to train an effective model. Unsupervised solutions as word embedding approaches can generate links by capturing the semantic meaning of artifacts and are gaining more attention. Despite that, our observation revealed that, besides the semantic information, the sequential information of terms in the artifacts would provide additional assistance for building the accurate links. This paper proposes an unsupervised requirements traceability approach (named S2Trace) which learns the Sequential Semantics of software artifacts to generate the trace links. Its core idea is to mine the sequential patterns and use them to learn the document embedding representation. Evaluation is conducted on five public datasets, and results show that our approach outperforms three typical baselines. The modeling of sequential information in this paper provides new insights into the unsupervised traceability solutions, and the improvement in the traceability accuracy further proves the usefulness of the sequential information.

Crowdtesting has become an effective alternative to traditional testing, especially for mobile applications. However, crowdtesting is hard to manage in nature. Given the complexity of mobile applications and unpredictability of distributed crowdtesting processes, it is difficult to estimate (a) remaining number of bugs yet to be detected or (b) required cost to find those bugs. Experience-based decisions may result in ineffective crowdtesting processes, e.g., there is an average of 32% wasteful spending in current crowdtesting practices. This paper aims at exploring automated decision support to effectively manage crowdtesting processes. It proposes an approach named ISENSE which applies incremental sampling technique to process crowdtesting reports arriving in chronological order, organizes them into fixed-size groups as dynamic inputs, and predicts two test completion indicators in an incremental manner. The two indicators are: 1) total number of bugs predicted with Capture-ReCapture model, and 2) required test cost for achieving certain test objectives predicted with AutoRegressive Integrated Moving Average model. The evaluation of ISENSE is conducted on 46,434 reports of 218 crowdtesting tasks from one of the largest crowdtesting platforms in China. Its effectiveness is demonstrated through two application studies for automating crowdtesting management and semi-automation of task closing trade-off analysis. The results show that ISENSE can provide managers with greater awareness of testing progress to achieve cost-effectiveness gains of crowdtesting. Specifically, a median of 100% bugs can be detected with 30% saved cost based on the automated close prediction.

In Continuous Integration (CI) development environment, integration testing (i.e., CI testing) is an important practice to verify the quality of submitted code. With the growth of integration system, running all tests leads to high test cost with slow feedback. Many test case selection techniques have been proposed to tackle this problem. We analyze existing static test selection approaches on method-level, and find there are two main drawbacks, i.e., omission of dependencies and imprecise dependencies, which influence the effectiveness of test selection. Both of them are due to the fact that some dependencies are dynamically determined at runtime, which could not be resolved solely by current static analysis techniques. We propose a Method-level tEst SelecTion approach (i.e., MEST), which utilizes static dependencies and dynamic execution rules (i.e., dynamic invocation in reflection and dynamic binding in inheritance). Evaluation is conducted on 18 open source projects with 261 continuous integration versions from Eclipse and Apache communities. We assess the effectiveness of MEST from reduced test size, fault detection efficiency and test cost, and compare it with the state-of-the-art approach ClassSRTS (as baseline); and further analyze the contribution of dynamic execution rules. Results show that (1) on average, MEST can reduce test size by 92% and 43% compared with actual CI testing and baseline respectively; (2) MEST can fully cover all faults detected by actual CI testing (in 97% versions) and baseline (in 98% versions), and find new faults in 26% and 27% versions respectively; (3) on average, the endto-end time of MEST is 24% and 48% of actual CI testing and baseline respectively; (4) both dynamic execution rules contribute to fault detection through capturing necessary dependencies. This approach can further speed up the feedback of CI testing and improve the fault detection efficiency of CI testing.

In open source communities, there are numerous duplicate issue reports, considered as useless and negligible by developers. Conversely, some researches argued that duplicates deliver complementary information that could benefit issue resolving. Considering all duplicates as value-neutral will result in either overestimation or underestimation of valuable information. It is necessary to be aware of whether all duplicates are redundant or beneficial. In this paper, we investigate whether duplicates have the same impacts on issue resolving and identification cost. We divide duplicates into three categories according to the statuses of master reports when duplicates are submitted. The results show duplicates in different categories play different roles in issue-resolving, and identification cost is also significantly different. Our study reveals duplicates are different, but almost are paid equal attentions. It is promising to propose new approaches and tools to resolve the problem.

Continuous Integration (CI) is an important practice in agile development.With the growth of integration system, running all tests to verify the quality of submitted code, is clearly uneconomical.This paper aims at selecting a proper test subset towards full coverage of all changed and affected code so as to reduce the cost of CI testing.We proposes FEST, a novel approach, which searches for the full dependencies of changed code at the class level and then selects test classes related to the changed and affected classes.We assess FEST from fault detection efficiency and cost effectiveness based on 18 open source projects with 261 continuous integration versions from Eclipse and Apache communities, and compare it with the stateof-the-art approach ClassSRTS (as baseline).Results show that FEST (1) can not only cover all faults detected by actual CI testing and baseline, but also find new faults in 25% and 18% versions respectively.(2) shows better or equal test scale benefits than actual CI testing (in 98% versions) and baseline (in 99% versions); and can compensate risk of omitting necessary tests for actual CI testing (in 62% versions) and baseline (in 73% versions).

Background: The most important challenge regarding the use of static analysis tools (e.g., FindBugs) is that there are a large number of warnings that are not acted on by developers. Many features have been proposed to build classification models for the automatic identification of actionable warnings. Through analyzing these features and related studies, we observe several limitations that make the users lack practical guides to apply these features.

Beyond the functional requirements of a system, software maintainability is essential for project success. While there exists a large knowledge base of software maintainability, this knowledge is rarely used in open source software due to the large number of developers and inefficiency in identifying quality issues. To effectively utilize the current knowledge base in practice requires a deeper understanding of how problems associated with the different qualities arise and change over time. In this paper, we sample over 6000 real bugs found from several Mozilla products to examine how maintainability is expressed with subgroups of repairability and modifiability. Furthermore, we manually study how these qualities evolve as the products mature, what the root causes of the bugs are for each quality and the impact and dependency of each quality. Our results inform which areas should be focused on to ensure maintainability at different stages of the development and maintenance process.

Kernel rootkits pose a serious threat to system security by tampering with the state of operating system inconspicuously. To ensure operating system kernel integrity, Virtual Machine Monitor (VMM) based approaches have been proposed. Most of these approaches use snapshot-based or event-triggered techniques. However, snapshot-based techniques have been suffering from missing transient attacks or significant performance overhead, while event-triggered methods are facing with heavy workload as integrity checking might be triggered by any suspicious actions. In this paper, we propose a novel solution which is a behavior-triggered integrity checking approach named BehaviorKI. By analyzing attacking processes, BehaviorKI can extract a set of behavior patterns which characterize malicious behaviors. BehaviorKI will trigger integrity checking with kernel invariants when a malicious behavior pattern detected. In this way, our approach can alleviate the performance burden by reducing the frequent kernel integrity checking. The experiment results show that Be-haviorKI outperforms existing snapshot-based and event-triggered approaches.

The development of High-level programming languages(HLPL) is a long process of evolution, which has gone through procedure-oriented languages, object-oriented languages, script languages and visual & database languages. Throughout the process of evolution, coding in an efficient and convenient way is the primary impetus. Hence, the productivity should vary across different languages. When evaluating the productivity of developers coding in different programming languages, such variations are usually ignored, which may not reflect their actual coding efficiencies and make the developers feel unfair. Especially, ignoring the variations will lead to inappropriate baselines of process performance, thus may potentially reduce the effectiveness of quantitative management. In this paper, we conducted an empirical study to revisit the productivity variations across different programming languages based on the data of International Software Benchmarking Standards Group (ISBSG) and a software organization. We found that, in most language categories, the productivity is all significantly different in ISBSG and the organization respectively. In the software industry represented by ISBSG, the relative productivity levels are gradually increasing with the evolution of HLPL. However, for the organization, it does not always keep the same increasing trend, but the average productivity within the four categories is almost stable. The finding could guide the software organization to establish an appropriate productivity baseline.

Crowdsourced testing is an emerging trend in software testing, which relies on crowd workers to accomplish test tasks. Due to the cost constraint, a test task usually involves a limited number of crowd workers. Furthermore, more workers does not necessarily result in detecting more bugs. Different workers, who may have different testing experience and expertise, may make much differences in the test outcomes. For example, some inappropriate workers may miss true bug, introduce false bugs or report duplicated bugs, which decreases the test quality. In current practice, a test task is usually dispatched in a random manner, and the quality of testing cannot be guaranteed. Therefore, it is important to select an appropriate subset of workers to perform a test task to ensure high bug detection rate. This paper introduces ExReDiv, a novel hybrid approach to select a set of workers for a test task. It consists of three key strategies: the experience strategy selects experienced workers, the relevance strategy selects workers with expertise relevant to the given test task, the diversity strategy selects diverse workers to avoid detecting duplicated bugs. We evaluate ExReDiv based on 42 test tasks from one of the largest crowdsourced testing platforms in China, and the experimental results show its effectiveness.

Mobile app testing is challenging since each test needs to be executed in a variety of operating contexts including heterogeneous devices, various wireless networks and different locations. Crowdsourcing enables a mobile app test to be distributed as a crowdsourced task to leverage crowd workers to accomplish the test. However, high test quality and expected test context coverage are difficult to achieve in crowdsourced testing. Upon distributing a test task, mobile app providers neither know who to participate nor predict whether all the expected test contexts can be covered in the task. To address this problem, we put forward a novel research problem called Crowdsourced Testing Quality Maximization Under Context Coverage Constraint (Cocoon). Given a mobile app test task, our objective is to recommend a set of workers, from available crowd workers, such that the expected test context coverage and a high test quality can be achieved. We prove that the Cocoon problem is NP-Complete and then introduce two greedy approaches. Based on a real dataset from the largest Chinese crowdsourced testing platform, our evaluation shows the effectiveness and efficiency of the two approaches, which can be potentially used as online services in practice.

In open software development environment, a large number of feature requests with mixed quality are often posted by stakeholders and usually managed in issue tracking systems. Thoroughly understanding and analyzing the real intents that feature requests imply is a labor-intensive and challenging task. In this paper, we introduce an approach to understand feature requests automatically. We generate a set of fuzzy rules based on natural language processing techniques that classify each sentence in feature requests into a set of categories: Intent, Explanation, Benefit, Drawback, Example and Trivia. Consequently, the feature requests can be automatically structured based on the classification results. We conduct experiments on 2,112 sentences taken from 602 feature requests of nine popular open source projects. The results show that our method can reach a high performance on classifying sentences from feature requests. Moreover, when applying fuzzy rules on machine learning methods, the performance can be improved significantly.

Links between issue reports and corresponding fix commits are widely used in software maintenance. The quality of links directly affects maintenance costs. Currently, such links are mainly maintained by error-prone manual efforts, which may result in missing links. To tackle this problem, automatic link recovery approaches have been proposed by building traditional classifiers with positive and negative links. However, these traditional classifiers may not perform well due to the inherent characteristics of missing links. Positive links, which can be used to build link recovery model, are quite limited as the result of missing links. Since the construction of negative links depends on the number of positive links in many existing approaches, the available negative links also become restricted. In this paper, we point out that it is better to consider the missing link problem as a model learning problem by using positive and unlabeled data, rather than the construction of traditional classifier. We propose PULink, an approach that constructs the link recovery model with positive and unlabeled links. Our experiment results show that compared to existing state-of-the-art technologies built on traditional classifier, PULink can achieve competitive performance by utilizing only 70% positive links that are used in those approaches.

Crowdsourced testing is an emerging trend in software testing, which relies on crowd workers to accomplish test tasks.Typically, a crowdsourced testing task aims to detect as many bugs as possible within a limited budget.For a specific test task, not all crowd workers are qualified to perform it, and different test tasks require crowd workers to have different experiences, domain knowledge, etc. Inappropriate workers may miss true bugs, introduce false bugs, or report duplicated bugs, which could not only decrease the quality of test outcomes, but also increase the cost of hiring workers.Thus, how to select the appropriate crowd workers for specific test tasks is a challenge in crowdsourced testing.This paper proposes a Multi-Objective crowd wOrker SElection approach (MOOSE), which includes three objectives: maximizing the coverage of test requirement, minimizing the cost, and maximizing bug-detection experience of the selected crowd workers.Specifically, MOOSE leverages NSGA-II, a widely used multi-objective evolutionary algorithm, to optimize the three objectives when selecting workers.We evaluate MOOSE on 42 test tasks (involve 844 crowd workers and 3,984 test reports) from one of the largest crowdsourced testing platforms in China, and the experimental results show MOOSE could improve the best baseline by 17% on average in bug detection rate.

Background: Links between issue reports and their fixing commits play an important role in software maintenance. Such link data are often missing in practice and many approaches have been proposed in order to recover them automatically. Most of existing approaches focus on comparing log messages and source code files in commits with issues reports. Besides the two kinds of data in commits, non-source documents (NSDs) such as change logs usually record the fixing activities and sometimes share similar texts as those in issue reports. However, few discussions have been made on the role of NSDs in designing link recovery approaches.

Context: Automatic classification of crowdsourced test reports is important due to their tremendous sizes and large proportion of noises. Most existing approaches towards this problem focus on examining the performance of different machine learning or information retrieval techniques, and most are evaluated on open source dataset. However, our observation reveals that these approaches generate poor and unstable performances on real industrial crowdsourced testing data. We further analyze the deep reason and find that industrial data have significant local bias, which degrades existing approaches.

Open Source Software (OSS) is distributed and maintained collaboratively by developers all over the world. However, frequent personnel turnover and lack of organizational management makes it difficult to capture the actual development effort. Various OSS maintenance effort estimation approaches have been developed to provide a way to understand and estimate development effort. The goal of this study is to identify the current state of art of the existing maintenance effort estimation approaches for OSS. We performed a systematic literature review on the relevant studies published in the period between 2000-2015 by both automatic and manual searches from different sources. We derived a set of keywords from the research questions and established selection criteria to carefully choose the papers to evaluate. 29 out of 3,312 papers were selected based on a well designed selection process. Our results show that the commonly used OSS maintenance effort estimation methods are actual effort estimation and maintenance activity time prediction, the most commonly used metrics and factors for actual effort estimation are source code measurements and people related metrics, the most commonly mentioned activity for maintenance activity time prediction is bug fixing. Accuracy measures and cross validation is used for validating the estimation models. Based on the above findings, we identified the issues in evaluation methods for actual maintenance effort estimations and the needs for quantitative OSS maintenance effort inference from size-related metrics. Meanwhile, we highlighted individual contribution and performance measurement as a novel and promising research area.

In crowdsourced testing, an important task is to identify the test reports that actually reveal fault - true fault, from the large number of test reports submitted by crowd workers. Most existing approaches towards this problem utilized supervised machine learning techniques, which often require users to manually label a large amount of training data. Such process is time-consuming and labor-intensive. Thus, reducing the onerous burden of manual labeling while still being able to achieve good performance is crucial. Active learning is one potential technique to address this challenge, which aims at training a good classifier with as few labeled data as possible. Nevertheless, our observation on real industrial data reveals that existing active learning approaches generate poor and unstable performances on crowdsourced testing data. We analyze the deep reason and find that the dataset has significant local biases. To address the above problems, we propose LOcal-based Active ClassiFication (LOAF) to classify true fault from crowdsourced test reports. LOAF recommends a small portion of instances which are most informative within local neighborhood, and asks user their labels, then learns classifiers based on local neighborhood. Our evaluation on 14,609 test reports of 34 commercial projects from one of the Chinese largest crowdsourced testing platforms shows that our proposed LOAF can generate promising results. In addition, its performance is even better than existing supervised learning approaches which built on large amounts of labelled historical data. Moreover, we also implement our approach and evaluate its usefulness using real-world case studies. The feedbacks from testers demonstrate its practical value.

Open source projects rely on issue tracking systems such as JIRA or online forums to keep track of users' feedback, expectations and requested features. However, since users are not fully aware of existing features, when submitting new feature requests, redundant requests often appear in the new feature list. It is a waste of time and effort for project contributors to manually identify and reject them, especially in complex systems with many features. Our research is aiming to find a suitable solution to identify redundant feature requests in OSS projects. We have conducted a survey on a well-known Open Source community, Hibernate and gathered all of its feature requests up-to-date. Through studying and categorizing the characteristics of these feature requests, we have found that about 37% of the feature requests were rejected and the most common rejection reason was redundancy. Also we have found that it is very expensive to identify and resolve these redundant feature requests. In this paper, we have proposed our solution to automatically identify redundant feature requests through a Feature Tree Model along with a future research agenda.

The timely fixing of bugs is important to ensure software quality.In Open Source Software (OSS) development, behaviors of stakeholders impact the bug fixing process, especially the different stages respectively.However, most of the existing studies on impact factors of bug fixing time usually treat bug fixing process as a whole, while neglecting the particularity at its different stages.Ignoring the detail of different stages cannot let us understand why the fixing time is longer or shorter.In this paper, we aimed at investigating whether the factors have different impacts on the time of different stages and the whole process.Three stages of the whole fixing process were formalized, and twenty-four factors were defined and extracted from three aspects: bug reports, their associated source code and code changes.An empirical study based on two OSS projects, Eclipse JDT Core and Linux Kernel, was conducted for the investigation.The results of our study provide a very positive validation that the influence of factors on bug fixing time is stage related, rather than for the whole process.Our results can help developers better understand influences of factors on the bug fixing process, and thus provide opportunities to improve their process effectively.

Studying the spread of phenomena in social networks is critical but still not fully solved. Existing influence maximization models assume a static network, disregarding its evolution over time. We introduce the continuous time constrained influence maximization problem for dynamic diffusion networks, based on a novel diffusion model called DynaDiffuse. Although the problem is NP-hard, the influence spread functions are monotonic and submodular, enabling fast approximations on top of an innovative stochastic model checking approach. Experiments on real social network data show that our model finds higher quality solutions and our algorithm outperforms state-of-art alternatives.

Context: Bug triage aims to recommend appropriate developers for new bugs in order to reduce time and effort in bug resolution. Most previous approaches for bug triage are supervised. Before recommending developers, these approaches need to learn developers' bug-fix preferences via building and training models using text-information of developers' historical bug reports.

Agile software engineering increasingly seeks to incorporate design modification and continuous refactoring in order to maintain code quality even in highly dynamic environments. However, there does not currently appear to be an industry-wide consensus on how to do this and research in this area expresses conflicting opinions. This paper presents an empirical study based upon an industry survey aimed at understanding the different ways that refactoring is thought of by the different people carrying out different roles in agile processes and how these different people weigh the importance of refactoring versus other kinds of tasks in the process. The study found good support for the importance of refactoring, but most respondents agreed that deferred refactoring impacts the agility of their process. Thus there was no universally agreed-upon strategy for planning refactoring. The survey findings also indicated that different roles have different perspectives on the different kinds of tasks in an agile process although all seem to want to increase the priority given to refactoring during planning for the iterations in agile development. Analysis of the survey raised many interesting questions suggesting the need for a considerable amount of future research.

Crowdsourcing is a distributed problem-solving and production model. It takes advantage of the internet technology, helps enterprises save cost and improve efficiency. However, uncertain quality is a significant challenge for crowdsourcing. On the basis of the existing literatures, this paper proposes 23 software quality factors from two aspects: platform and project. By using multiple regression analysis on the data of one of the most successful software crowdsourcing platforms TopCoder.com, this paper analyzes the impact of the factors on software quality and identifies six key factors, including the average quality score of the platform, the number of contemporary projects, the length of component document, the number of registered developers, the maximum rating of submitted developers, and the design score. According to the result, this paper suggests four aspects for enterprises to improve software quality: choosing the prosperous period of platform to post a project, reducing the scale of projects, attracting more and higher skillful developers to participate, and improving software design score.

During open source software development and maintenance, bug fixing is a result of developer collaboration. Understanding the structure of developer collaboration could be helpful for effective and efficient bug fixing. Most prior work on exploring developer collaboration in bug repositories only considers a particular form of developer collaboration. However, in real software bug repositories, developers collaborate with each other via multiple ways, e.g., commenting bugs, tossing bugs, and assigning bugs. In this paper, we present DevNet, a framework for representing and analyzing developer collaboration in bug repositories based on heterogeneous developer networks. Moreover, we illustrate that such developer collaboration can assist bug triage through a case study on the bug repositories of Eclipse and Mozilla involving over 800,000 bug reports. Experiment results show that our approach can improve the state-of-the-art bug triage methods by 5-15% in accuracy. We believe that the proposed approach provides new insights for analyzing software repositories through heterogeneous networks.

Software projects are always facing various risks. These risks should be identified, analyzed, prioritized, mitigated, monitored and controlled. After risks are identified and analyzed, resources must then be devoted to mitigation. However, risk prioritization and mitigation planning are complicated problems. Especially in project portfolio management (PPM), resource contention among projects leads to difficulty in choosing and executing mitigation actions. This paper introduces a search based risk mitigation planning method that is useful in PPM. It integrates the analysis of risks, consideration of available resources, and evaluation of possible effects when taking risk mitigation actions. The method uses a genetic algorithm to search for the risk mitigation plan of optimal value. A case study shows how this method can identify effective risk mitigation plans, thus providing useful decision support for managers.

Managing the costs and risks of evolution is a challenging problem in the RE community. The challenge lies in the difficulty of analyzing and assessing the proneness to requirement changes across multiple versions, especially when the scale of requirements is large. In this paper, we define a series of metrics to characterize historic evolution information, and propose a novel method for predicting requirements that are likely to evolve in the future based on the metrics. We apply the prediction method to analyze the product updates history through a case study. The empirical results show that this method can provide a tradeoff solution that narrows down the scope of change analysis to a small set of requirements, but it still can retrieve nearly half of the future changes. The results indicate that the defined metrics are sensitive to the history of requirements evolution, and the prediction method can reach a valuable outcome for requirement engineers to balance their workload and risks.

Complexity cohesion and coupling have been recognized as prominent indicators for software quality. One characterization of software complexity is the existence of dependency relationship. Moreover, degree of dependency reflects the cohesion and coupling between software elements. Dependencies on design and implementation phase have been proven as important predictors for software bugs. We empirically investigated how requirements dependencies correlate with and predict software integration bugs, which can provide early estimate regarding software quality, therefore facilitate decision making early in the software lifecycle. We conducted network analysis on requirements dependency networks of two commercial software projects. We then performed correlation analysis between network measures (e.g., degree, closeness) and number of bugs. Afterwards, bug prediction models were built using these network measures. Significant correlation is observed between most of our network measures and number of bugs. These network measures can predict the number of bugs with high accuracy and sensitivity. We further identified the significant predictors for bug prediction. Besides, the indication effect of network measures on bug number varies among different types of requirements dependency. These observations show that requirements dependency network can be used as an early indicator of software Integration bugs.

Background: Change propagation analysis helps predict the parts of the software that may be affected if a change is made. Existing research on change propagation focuses on design and code level changes. However, as a software evolves, the requirements that drive these changes also have intricate dependencies. Understanding the effect of these requirement dependencies on change prorogation is useful but not trivial. More than twenty requirements dependency types have been identified in the literature, however there still lacks an evaluation of the applicability of these dependency types in requirements and change propagation analysis. Aim: We aim to investigate whether these dependency types are useful for change propagation analysis. Method: We conducted a case study in a real-world industry project. This case study evaluates two representative dependency models covering twenty five types of dependencies. Results: Our initial evaluation has found that five dependency types are particularly useful in change propagation analysis and practitioners with different backgrounds have various viewpoints on change propagation. Thus change impact analysis should involve a wide range of stakeholders including project managers, requirements engineers, designers and developers. Conclusions: Our case study provides insights into requirements dependencies and their effects on change propagation analysis for both research and practice.

Background: Software systems must evolve in order to adapt in a timely fashion to the rapid changes of stakeholder needs, technologies, business environment and society regulations. Numerous studies have shown that cost, schedule or defect density of a software project may escalate as the requirements evolve. Requirements evolution management has become one important topic in requirements engineering research. Aim: To depict a holistic state-of-the-art of requirement evolution management. Method: We undertook a systematic review on requirements evolution management. Results: 125 relevant studies were identified and reviewed. This paper reports the preliminary results from this review: (1) the terminology and definition of requirements evolution; (2) fourteen key activities in requirements evolution management; (3) twenty-eight metrics of requirements evolution for three measurement goals. Conclusions: Requirements evolution is a process of continuous change of requirements in a certain direction. Most existing studies focus on how to deal with evolution after it happens. In the future, more research attention on exploring the evolution laws and predicting evolution is encouraged.

has become an important software engineering research method but costs tremendous efforts. [Aim]: This paper proposes an approach to leverage on empirically evolved ontology to support automating key SLR activities. [Method]: First, we propose an ontology, SLRONT, built on SLR experiences and best practices as a groundwork to capture common terminologies and their relationships during SLR processes; second, we present an extended version of SLRONT, the COSONT and instantiate it with the knowledge and concepts extracted from structured abstracts. Case studies illustrate the details of applying it for supporting SLR steps. [Results]: Results show that through using COSONT, we acquire the same conclusion compared with sheer manual works, but the efforts involved is significantly reduced. [Conclusions]: The approach of using ontology could effectively and efficiently support the conducting of systematic literature review. Keywords-systematic literature review; ontology; structured abstract; software cost estimation I.

Software effort data contains a large amount of missing values of project attributes. The problem of absent features, which occurred recently in machine learning, is often neglected by researchers of software engineering when handling the missingness in software effort data. In essence, absent features (structural missingness) and unobserved values (unstructured missingness) are different cases of missingness although their appearance in the data set are the same. This paper attempts to clarify the root cause of missingness of software effort data. When regarding missingness as absent features, we develop Max-margin regression to predict real effort of software projects. When regarding missingness as unobserved values, we use existing imputation techniques to impute missing values. Then, ε – SVR is used to predict real effort of software projects with the input data sets. Experiments on ISBSG (International Software Benchmarking Standard Group) and CSBSG (Chinese Software Benchmarking Standard Group) data sets demonstrate that, with the tasks of effort prediction, the treatment regarding missingness in software effort data set as unobserved values can produce more desirable performance than that of regarding missingness as absent features. This paper is the first to introduce the concept of absent features to deal with missingness of software effort data.

SUMMARY For a successful software project, acceptable quality must be achieved within an acceptable cost, demonstrating business value to customers and satisfactorily meeting delivery timeliness. Testing serves as the most widely used approaches to determine that the intended functionalities are performed correctly and achieve the desired level of services; however, it is also a labor‐intensive and expensive process during the whole software life cycle. Most current testing processes are often technique‐centered, rather than organized to maximize business value. In this article, we extend and elaborate the ‘4+1’ theoretical lenses of Value‐based Software Engineering (VBSE) framework in the software testing process; propose a multi‐objective feature prioritization strategy for testing planning and controlling, which aligns the internal testing process with value objectives coming from customers and markets. Our case study in a real‐life business project shows that this method allows reasoning about the software testing process in different dimensions: it helps to manage the testing process effectively and efficiently, provides information for continuous internal software process improvement, and increases customer satisfaction, which makes winners of all success‐critical stakeholders (SCSs) in the software testing process. Copyright © 2010 John Wiley & Sons, Ltd.

Selection of the right user stories and planning their implementation for the next iteration is critical for success of extreme Programming (XP). Success here is measured by the total business value generated from all user stories implemented within time. The business value of an iteration is composed by the value of the individual user stories selected and additional value created from themes of user stories. In this paper, a method combining advanced search and risk analysis is proposed to support decision-making for the "best" set of user stories. The advanced search technique combines genetic search with subsequent application of the hill climbing technique. The top candidate solutions are further analyzed pro-actively in terms of their risk to be implementable in-time with the available effort. As a proof-of-concept, the applicability of the proposed method is applied for the planning of one iteration in a case study project with 40 user stories. As a result, a set of trade-off solutions is offered as decision support for XP teams.

This paper proposes a new approach called DREX (Developer Recommendation with k-nearest-neighbor search and Expertise ranking) to developer recommendation for bug resolution based on K-Nearest-Neighbor search with bug similarity and expertise ranking with various metrics, including simple frequency and social network metrics. We collect Mozilla Fire fox open bug repository as the experimental data set and compare different ranking metrics on the performance of recommending capable developers for bugs. Our experimental results demonstrate that, when recommending 10 developers for each one of the 250 testing bugs, DREX has produced better performance than traditional methods with multi-labeled text categorization. The best performance obtained by two metrics as Out-Degree and Frequency, is with recall as 0.6 on average. Moreover, other social network metrics such as Degree and Page Rank have produced comparable performance on developer recommendation as Frequency when used for developer expertise ranking.

Testing is one of the most labor-intensive activities in software development life cycle and consumes between 30% and 50% of total development costs according to many studies. The communication gap between testers and developers that is caused by unclear or even invalid defect reporting usually makes the testing schedule delay, and contributes large amount of testing effort to rework and re-communication. Defect reporting's efficiency directly influences testing efficiency and effectiveness. From our observation and improvement practices on a software product with recent 4 increments: 1) the percentage of unclear defect reports is around 23%, but more than 82% testers, more or less, reported unclear defect reports, 2) the percentage of invalid defect reports is around 26%, but more than 35% testers reported invalid defect reports. Improvement practices that we took on training testers on how to report defect in a clearer and more efficient way further bridged the communication gap between testers and developers, and it can reduce 15.7% unclear defect reports and 19.53% invalid defect reports, which finally led to the improvement of testing efficiency.

More and more software applications are developed within a software ecosystem (SECO), such as the Face book ecosystem and the iPhone AppStore. A core asset of a software ecosystem is its users, and the behavior of the users strongly affects the decisions of software vendors. The number of active users reflects user satisfaction and quality of the applications in a SECO. However, we can hardly find any literature about the number of active software users. Because software users are one of the most important assets of a software business, this information is very sensitive. In this paper, we analyzed the traces of software application users within a large scale software ecosystem with millions of active users. We identified useful patterns of user behavior, and proposed models that help to understand the number of active application users. The model we proposed better predicts the number of active users than just looking at the traditional retention rate. It also provides a fast way to monitor user satisfaction of online software applications. We have therefore provided an alternative way for SECO platform vendors to identify rising or falling applications, and for third party application vendors to identify risks and opportunity of their products.

Software maintenance effort constitutes a major portion of the software lifecycle effort. Its estimation is vital for successful project planning and strategic resource allocation. In this paper, we conduct and report an industrial case study in this field. The data set was collected from an industrial software process management tool QONE (formerly SoftPM). The methodology proposed provides corresponding guidance for effort estimation in software evolutionary projects that employ use-cases in capturing maintenance requirements. And the model, constructed using the linear regression analysis and validated by the leave-one-out cross-validation, provides an effort prediction for the future maintenance of the project. The analysis results indicate that the methodology can be applied at an early stage of the project life cycle and provides a good tradeoff among simplicity, early-estimating and accuracy in one estimate.

As a new contribution to Value-based V&V process development, a systematic and multi-criteria process is proposed to quantitatively determine the Value-based V&V artifact priority that reviewers can follow for their reviews. This process enables reviewers to prioritize artifacts to be reviewed in a more cost-effective way based on more sophisticated and comprehensive factors, such as importance, quality risks, dependency and cost of V&V investments. Some qualitative and quantitative evidence is provided from a comparative experiment with 22 real-client e-services projects over two years of a graduate software engineering team-project course. It shows that the value-based artifact prioritization enabled reviewers to better focus on artifacts with high importance and risks, to capture issues with high impact in a timely manner, and to improve the cost-effectiveness of reviews.

Nowadays, commercial or in-house customized process management tools have been prevalently adopted for supporting software project management and process improvement. In this paper we report a case study to empirically investigate and evaluate the usage status and implications of a supporting tool named QONE in industrial environment in China. Decision theory is adopted in study design. The analysis is mainly based on the usage data from a typical industrial project. Further questionnaires and follow-up interviews with the end-users are also conducted. The analysis results reveal that 1) the effects of such supporting tool vary with respect to different task types, 2) tasks with smaller granularity are comparatively easier to predict and control, 3) missing data reporting analysis helps to reveal opportunities for further process improvement and tool enhancement. This investigation aims to help us take advantages of such supporting tools and benefit software development eventually.

Requirements evolution is one of critical problems influencing software engineering activities. Despite there is much research on requirements evolution, there still lacks quantitative understanding of requirements evolution. In this paper, we quantitatively analyze requirements evolution across multiple versions of an industrial software product. Based on data of requirements evolution and defects, we analyze the relationship between requirements evolution and requirements as well as between defects and requirements evolution. We also analyze the evolution characteristics about requirements modification. Our findings include estimation of the number of defects using evolved requirements may increase accuracy of defect estimation and business rule is the most volatile part in requirements. These findings deepen our understanding of requirements evolution and can help software organizations manage requirements evolution.

Software testing is an important process for software quality control, and it is also costly and time-consuming. It can easily take 50% of development time [1]. To gain high quality software, a frequently-adopted approach is to try to minimize the number of defects in the final product, and it depends on software testing effectiveness. Many studies show that testing process is among one of the most difficult management practices. It is the last step to remove the defects before the software product to be released. But how to evaluate the quality of testing, a common way is to measure the defects detected during testing. When a defect was detected, if we can know the probability of the defect can be fixed, we can economize the limited time to fix more defects. But how we know the defect can be fixed or not before fixing. In our study, we want to find out which factors affect the defect removal.

Jeffery and Votta [1] consider that the central task of an empirical software engineering (ESE) study is the use of data to address research questions. And as the outcome of empirical studies, experience is reported. When more and more empirical studies are conducted, it becomes difficult to keep track of which data is used in which study, and what is the experience gained. In addition, similar studies generally use slightly different versions of a particular data source to support their own research methods, which can easily cause confusion. Another important problem is that experience and data are not easy to categorize, because different criteria produce different sets of categories. Therefore, there is an increasing need for better management of software data and experience in ESE studies.

To address business requirements and to survive in competing markets, companies or open source organizations often have to release different versions of their projects in different languages. Manually migrating projects from one language to another (such as from Java to C#) is a tedious and error-prone task. To reduce manual effort or human errors, tools can be developed for automatic migration of projects from one language to another. However, these tools require the knowledge of how Application Programming Interfaces (APIs) of one language are mapped to APIs of the other language, referred to as API mapping relations. In this paper, we propose a novel approach, called MAM (Mining API Mapping), that mines API mapping relations from one language to another using API client code. MAM accepts a set of projects each with two versions in two languages and mines API mapping relations between those two languages based on how APIs are used by the two versions. These mined API mapping relations assist in migration of projects from one language to another. We implemented a tool and conducted two evaluations to show the effectiveness of MAM. The results show that our tool mines 25,805 unique mapping relations of APIs between Java and C# with more than 80% accuracy. The results also show that mined API mapping relations help reduce 54.4% compilation errors and 43.0% defects during migration of projects with an existing migration tool, called Java2CSharp. The reduction in compilation errors and defects is due to our new mined mapping relations that are not available with the existing migration tool.

Resource allocation in software development is important and many methods have been proposed. Related empirical research is yet scarce and evidence is required to validate the theoretical methods. This paper introduces the staffing pattern as a metric of resource distribution among project phases, and verifies its effect on software quality and productivity using real project data. The main findings are: (1) there exist different staffing patterns in reality; (2) the staffing pattern has significant effect on software quality (post-release defect density); (3) the staffing pattern has no significant effect on productivity; (4) the effort invested on test, document or code inspection possibly explains the effect of staffing pattern on software quality; (5) the effort consumed by rework perhaps counteracts the effect of other potential factors on productivity. Preliminary heuristics are suggested to resource allocation practices.

Bug assignment is an important step in bug life-cycle management. In large projects, this task would consume a substantial amount of human effort. To compare with the previous studies on automatic bug assignment in FOSS (free/open source software) projects, we conduct a case study on a proprietary software project in China. Our study consists of two experiments of automatic bug assignment, using Chinese text and the other non-text information of bug data respectively. Based on text data of the bug repository, the first experiment uses SVM to predict bug assignments and achieve accuracy close to that by human triagers. The second one explores the usefulness of non-text data in making such prediction. The main results from our study includes that text data are most useful data in the bug tracking system to triage bugs, and automation based on text data could effectively reduce the manual effort.

Nowadays software projects are still suffering from many problems due to various kinds of software risks. Software risk management is a crucial part of successful project management, but it is often not well implemented in real-world software projects. One reason is that project managers lack effective and practical tools to manage software risks. Software process simulation modeling (SPSM) has been emerging as a promising approach to address a variety of issues in software engineering area, including risk management. However, the current state of how SPSM supports software risk management is not yet clear. This paper presents a systematic literature review which purpose is to obtain the state of the art of the applications of SPSM in software risk management. We drew the following conclusions from the review results: (1) The number of SPSM studies on software risk management is relatively small, but increasing gradually in recent years. (2) SPSM is mainly applied in risk analysis and risk management planning activities. (3) Software risks related to requirements, development process and management process are the ones most studied by SPSM. (4) Discrete-event simulation and system dynamics are two most popular simulation paradigms, while Hybrid simulation methods are more and more widely used. (5) Extend, iThink and Vensim are the most popular simulation tools in SPSM. (6) Most of SPSM approaches and models have not been well applied into real-world risk management practices.

As two major performance measures, software productivity and quality convey critical information in supporting many decision making situations during project planning, management, as well as process/organization benchmarking processes. However, there is a lack of investigation on these performance measures with respect to enhancement projects, this leads to, in many enhancement/maintenance cases, the inappropriate application of techniques or benchmarking data resulted from studies using development project data. In this paper, through analysis of 264 enhancement projects in China, we seek to develop in-depth and comprehensive understanding about software enhancement projects, by examining the variance of productivity and defect density by several significant influencing factors, such as business area, region, language, programming tool, project size, and team size.

Software effort estimation techniques abound, each with its own set of advantages and disadvantages, and no one proves to be the single best answer. Combining estimating is an appealing approach. Avoiding the difficult problem of choosing the single "best" technique, it solves the problem by asking which techniques would help to improve accuracy, assuming that each has something to contribute. In this paper, we firstly introduce the systematic "external" combining idea into the field of software effort estimation, and estimate software effort using Optimal Linear Combining (OLC) method with an experimental study based on a real-life data set. The result indicates that combining different techniques can significantly improve the accuracy and consistency of software effort estimation by making full use of information provided by all components, even the much "worse" one.

To date, very little published empirical data has reported on the quality and reliability aspects of commercial software systems. In this paper, we present quantitative empirical study results on faults and failures with four releases of SoftPM [29, 31], one of the most widely adopted software project management tools in China. Our approach verifies Fenton's Hypotheses in [11] and explores the relationship between pre-release faults and post-release failures in four releases of SoftPM. We also present the distribution of Mean Time to Remove Faults (MTRF), the review and testing efficiency, and the fault and failure data for benchmarking. Our study shows that a few modules that are not fault-prone contain most failures in the field and most faults detected at an early stage cost less time to be fixed and removed. Our results have been validated and evaluated at four subsequent major releases of SoftPM at the laboratory for Internet Software Technologies of the Institute of Software, Chinese Academy of Sciences (iTechs).

Effort distribution by phase or activity is an important but often overlooked aspect compared to other steps in the cost estimation process. Poor effort allocation is among the major root causes of rework due to insufficiently resourced early activities. This paper provides results of an empirical study on phase effort distribution data of 75 industry projects, from the China Software Benchmarking Standard Group (CSBSG) database. The phase effort distribution patterns and variation sources are presented, and analysis results show some consistency in effects of software size and team size on code and test phase distribution variations, and some considerable deviations in requirements, design, and transition phases, compared with recommendations in the COCOMO model. Finally, this paper discusses the major findings and threats to validity and presents general guidelines in directing effort allocation. Empirical findings from this study are beneficial for stimulating discussions and debates to improve cost estimation and benchmarking practices.

Although a lot of attention has been paid to software cost estimation since 1960, making accurate effort and schedule estimation is still a challenge. To collect evidence and identify potential areas of improvement in software cost estimation, it is important to investigate the estimation accuracy, the estimation method used, and the factors influencing the adoption of estimation methods in current industry. This paper analyzed 112 projects from the Chinese software project benchmarking dataset and conducted questionnaire survey on 116 organizations to investigate the above information. The paper presents the current situations related to software project estimation in China and provides evidence-based suggestions on how to improve software project estimation. Our survey results suggest, e.g., that large projects were more prone to cost and schedule overruns, that most computing managers and professionals were neither satisfied nor dissatisfied with the project estimation, that very few organizations (15%) used model-based methods, and that the high adoption cost and insignificant benefit after adoption were the main causes for low use of model-based methods.

Software processes are highly people-dependent and knowledge transfer-centric compared to traditional manufacturing processes. Different people are responsible for different types of knowledge transformation according to the skill set and expertise they master. This adds a great deal of complicated factors in resolving the scheduling problem in software development and maintenance process planning. The existing human resource scheduling methods do not take into account the differences between human resource capabilities and capacities in processes execution. This paper presents a constraint-driven human resource scheduling method in software development and maintenance process. A constraint model is set up based on the software process model and human resource model. A constraint-driven scheduling method is provided to realize the optimal human resource scheduling in software development and maintenance process. The method can be used in the mature organizations whose human resources have the determinate capabilities. It provides the excellent decision support to the project manager.

Distribution of effort in software engineering process has been the basis for facilitating more reasonable software project planning. This paper reports empirical results on activity effort distribution patterns of a series of industrial software maintenance projects. The results show that with respect to different influencing factors, the projects demonstrate large variations in their activity effort distribution, which necessitates appropriate adjustments to strategic planning.

In software projects it is crucial to control the actual project schedule, cost and product quality, against the project plan. Variations often occur during the running project, and big ones might have significant impacts on the whole project. Thus, it is necessary to identify causes quickly, and take the right corrective actions in time. This paper provides a threedimensional software development model, called AHA (Activity-Human-Artifact). It combines three key factors, activity, human and artifact, corresponding to project schedule, cost, and product quality, respectively. We also define some relevant elements, attributes and relationships among different dimensions in the AHA model, and their impacts with respect to schedule, cost and quality. On the basis of our model, a project tracking process is developed to guide analysts to locate potential causes for different kinds of variations, and effectively correct them in real projects. We also take a software project as the application scenario to illustrate our model and the tracking process identified.

Productivity is a critical performance index of process resources. As successive history productivity data tends to be auto-correlated, time series prediction method based on auto-regressive integrated moving average (ARIMA) model was introduced into software productivity prediction by Humphrey et al. In this paper, a variant of their prediction method named ARIMAmmse is proposed. This variant formulates the ARIMA parameter estimation issue as a minimum mean square error (MMSE) based constrained optimization problem. The ARIMA model is used to describe constraints of the parameter estimation problem, while MMSE is used as the objective function of the constrained optimization problem. According to the optimization theory, ARIMAmmse will definitely achieve a higher MMSE prediction precision than Humphrey et al's which is based on the Yule-Walk estimation technique. Two comparative experiments are also presented. The experimental results further confirm the theoretical superiority of ARIMAmmse

High-level process management is quantitative management. The Process Performance Baseline (PPB) of process or subprocess under statistical management is the most important concept. It is the basis of process control and improvement. The existing methods for establishing process baseline are too coarse-grained or have some limitation, which lead to inaccurate or ineffective quantitative management. In this paper, we propose an approach called BSR (Baseline-Statistic-Refinement) for establishing and refining software process performance baseline, and present the experience result to validate its effectiveness for quantitative process management.