Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by retrieving relevant documents from external corpora before generating responses. This approach significantly expands LLM capabilities by leveraging vast, up-to-date external knowledge. However, this reliance on external knowledge makes RAG systems vulnerable to corpus poisoning attacks that manipulate generated outputs via poisoned document injection. Existing poisoning attack strategies typically treat the retrieval and generation stages as disjointed, limiting their effectiveness. We propose Joint-GCG, the first framework to unify gradient-based attacks across both retriever and generator models through three innovations: (1) Cross-Vocabulary Projection for aligning embedding spaces, (2) Gradient Tokenization Alignment for synchronizing token-level gradient signals, and (3) Adaptive Weighted Fusion for dynamically balancing attacking objectives. Evaluations demonstrate that Joint-GCG achieves at most 25% and an average of 5% higher attack success rate than previous methods across multiple retrievers and generators. While optimized under a white-box assumption, the generated poisons show unprecedented transferability to unseen models. Joint-GCG's innovative unification of gradient-based attacks across retrieval and generation stages fundamentally reshapes our understanding of vulnerabilities within RAG systems.

Jie Huang, Xin Liao, Junjie Wang, Mingyang Li, Wenshuo Wang, Ziyou Jiang, Shoubin Li, Qing Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Ziyou Jiang, Mingyang Li, Junjie Wang, Yuekai Huang, Jie Huang, Zhiyuan Chang, Zhaoyang Li, Qing Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Visual entailment (VE) is a multimodal reasoning task consisting of image-sentence pairs whereby a promise is defined by an image, and a sentence describes a hypothesis. The goal is to predict whether the image semantically entails the sentence. VE systems have been widely adopted in many downstream tasks such as image caption and visual question answering. However, the robustness of VE systems still faces significant challenges. One of the reasons is that the VE system suffers object-confusing defect when some similar objects exist. It outputs a positive prediction inferred by an erroneous object relationship, which will result in a fault negative prediction if the noised object does not exist. Previous approaches generate tests primarily relied on some general perturbations, such as simulating noise or weather interference in images, or substituting synonyms or rewriting sentences in texts. To test the object-confusing defect in VE systems, it requires perceiving and understanding key objects and entities and maintain the semantic relevance between cross-modal inputs, making it challenging to generate effective tests with high quality. Therefore, we propose VEglue , an object-aligned joint erasing approach for VE systems testing. It first aligns the object regions in the premise and object descriptions in the hypothesis to identify linked and un-linked objects. Then, based on the alignment information, three metamorphic relations are designed to jointly erase the objects of the two modalities. We evaluate VEglue on four widely used VE systems involving two public datasets, and the results demonstrate that VEglue could detect 11,609 issues on average with a 52.5% Issue Finding Rate (IFR). Furthermore, we leverage the tests generated by VEglue to retrain the VE systems, which largely improves model performance (50.8% increase in accuracy) on newly generated tests without sacrificing the accuracy on the original test set.

Information theft attacks pose a significant risk to Large Language Model (LLM) tool-learning systems. Adversaries can inject malicious commands through compromised tools, manipulating LLMs to send sensitive information to these tools, which leads to potential privacy breaches. However, existing attack approaches are black-box oriented and rely on static commands that cannot adapt flexibly to the changes in user queries and the invocation chain of tools. It makes malicious commands more likely to be detected by LLM and leads to attack failure. In this paper, we propose AutoCMD, a dynamic attack comment generation approach for information theft attacks in LLM tool-learning systems. Inspired by the concept of mimicking the familiar, AutoCMD is capable of inferring the information utilized by upstream tools in the toolchain through learning on open-source systems and reinforcement with target system examples, thereby generating more targeted commands for information theft. The evaluation results show that AutoCMD outperforms the baselines with +13.2% ASR_{Theft}, and can be generalized to new tool-learning systems to expose their information leakage risks. We also design four defense methods to effectively protect tool-learning systems from the attack.

Rupeng Zhang, Haowei Wang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, Qing Wang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Internet technology has given rise to an open-collaborative software development paradigm, necessitating the open-collaborative schema to software testing. It enables diverse and globally distributed contributions, but also presents significant challenges to efficient testing processes, coordination among personnel, and management of testing artifacts. At the same time, advancements in AI have enhanced testing capabilities and enabling automation, while also introducing new testing needs and unique challenges for AI-based systems. In this context, this article explores software testing in the open-collaborative and AI-powered era, focusing on the interrelated dimensions of process, personnel, and technology. Among them, process involves managing testing workflows and artifacts to improve efficiency, personnel emphasizes the role of individuals in ensuring testing quality through collaboration and contributions, while technology refers to AI methods that enhance testing capabilities and address challenges in AI-based systems. Furthermore, we delve into the challenges and opportunities arising from emerging technologies such as Large Language Models (LLMs) and the AI model-centric development paradigm.

Crash reports play a crucial role in software maintenance as they inform developers about the issues encountered in mobile applications. Developers must reproduce the reported crash before fixing it, which is extremely time-consuming and tedious. Existing studies have focused on automatic crash reproduction with step-by-step instructions. However, a non-neglectable portion of crash reports only provides a one-sentence overview, which merely describes the final crash-triggering action. These reports require developers to invest more effort in understanding and fixing the issues while existing techniques cannot handle them due to the lack of step-by-step guidance, thus calling for a greater need for automatic support. Leveraging the capability of Large Language Models (LLMs) in combining acting and reasoning, we propose ReActDroid, an automated approach to reproduce mobile application crashes directly from the crash overview. ReActDroid utilizes ReAct prompting to augment the app-specific knowledge and exploration history, enabling the LLM to derive the necessary steps for triggering the crash from a comprehensive and historical perspective. We evaluate ReActDroid on 102 crash reports from 69 popular Android apps and successfully reproduce 57.8% of the crashes, surpassing the performance of state-of-the-art baselines by 69% to 321%. Besides, the average reproducing time is 51.8 seconds, outperforming the baselines by 73% to 949%. We also evaluate the usefulness of ReActDroid with promising results.

Requirements linking is the core of software system maintenance and evolution, and it is critical to assuring software quality. In practice, however, the requirements links are frequently absent or incorrectly labeled, and reconstructing such ties is time-consuming and error-prone. Numerous learning-based approaches have been put forth to address the problem. However, these approaches will lose effectiveness for the Cold-Start projects with few labeled samples. To this end, we propose RADIATION, an adversarial-based domain adaptation approach for cross-domain requirements linking. Generally, RADIATION firstly adopts an IDF-based Masking strategy to filter the domain-specific features. Then it pre-trains a linking model in the source domain with sufficient labeled samples and adapts the model to target domains using a distance-enhanced adversarial technique without using any labeled target samples. Evaluation on five public datasets shows that RADIATION could achieve 66.4% precision, 89.2% recall, and significantly outperform state-of-the-art baselines by 13.4% -42.9% F1. In addition, the designed components, i.e., IDF-based Masking and Distance-enhanced Loss, could significantly improve performance.

Bug reports are vital for software maintenance that allow the developers being informed of the problems encountered in the software. Before bug fixing, developers need to reproduce the bugs which is an extremely time-consuming and tedious task, and it is highly expected to automate this process. However, it is challenging to do so considering the imprecise or incomplete natural language described in reproducing steps, and the missing or ambiguous single source of information in GUI components. In this paper, we propose a context-aware bug reproduction approach ScopeDroid which automatically reproduces crashes from textual bug reports for mobile apps. It first constructs a state transition graph (STG) and extracts the contextual information of components. We then design a multi-modal neural matching network to derive the fuzzy matching matrix between all candidate GUI events and reproducing steps. With the STG and matching information, it plans the exploration path for reproducing the bug, and enriches the initial STG iteratively. We evaluate the approach on 102 bug reports from 69 popular Android apps, and it successfully reproduces 63.7% of the crashes, outper-forming the state-of-the-art baselines by 32.6% and 38.3%. We also evaluate the usefulness and robustness of ScopeDroid with promising results. Furthermore, to train the neural matching network, we develop a heuristic-based automated training data generation method, which can potentially motivate and facilitate other activities as user interface operations.

Natural language (NL) documentation is the bridge between software managers and testers, and NL test cases are prevalent in system-level testing and other quality assurance activities. Due to reasons such as requirements redundancy, parallel testing, tester turn-over within long evolving history, there are inevitably lots of redundant test cases, which significantly increase the cost. Previous redundancy detection approaches typically treat the textual descriptions as a whole to compare their similarity and suffer from low precision. Our observation reveals that a test case can have explicit test-oriented entities, such as tested function Components, Constraints, etc; and there are also specific relations between these entities. This inspires us with a potential opportunity for accurate redundancy detection. In this paper, we first define five test-oriented entity categories and four associated relation categories, and re-formulate the NL test case redundancy detection problem as the comparison of detailed testing content guided by the test-oriented entities and relations. Following that, we propose Tscope, a fine-grained approach for redundant NL test case detection by dissecting test cases into atomic test tuple(s) with the entities restricted by associated relations. To serve as the test case dissection, Tscope designs a context-aware model for the automatic entity and relation extraction. Evaluation on 3,467 test cases from ten projects shows Tscope could achieve 91.8% precision, 74.8% recall and 82.4% F1, significantly outperforming state-of-the-art approaches and commonly-used classifiers. This new formulation of the NL test case redundant detection problem can motivate the follow-up studies in further improving this task and other related tasks involving NL descriptions.

Online chatting is gaining popularity and plays an increasingly significant role in software development. When discussing functionalities, developers might reveal their desired features to other developers. Automated mining techniques towards retrieving feature requests from massive chat messages can benefit the requirements gathering process. But it is quite challenging to perform such techniques because detecting feature requests from dialogues requires a thorough understanding of the contextual information, and it is also extremely expensive on annotating feature-request dialogues for learning. To bridge that gap, we recast the traditional text classification task of mapping single dialog to its class into the task of determining whether two dialogues are similar or not by incorporating few-shot learning. We propose a novel approach, named FRMiner, which can detect feature-request dialogues from chat messages via deep Siamese network. We design a BiLSTM-based dialog model that can learn the contextual information of a dialog in both forward and reverse directions. Evaluation on the real-world projects shows that our approach achieves average precision, recall and F1-score of 88.52%, 88.50% and 88.51%, which confirms that our approach could effectively detect hidden feature requests from chat messages, thus can facilitate gathering comprehensive requirements from the crowd in an automated way.

Requirement entities, "explicit specification of concepts that define the primary function objects", play an important role in requirement analysis for software development and maintenance. It is a labor-intensive activity to extract requirement entities from textual requirements, which is typically done manually. A few existing studies propose automated methods to support key requirement concept extraction. However, they face two main challenges: lack of domain-specific natural language processing techniques and expensive labeling effort. To address the challenges, this study presents a novel approach named RENE, which employs LSTM-CRF model for requirement entity extraction and introduces the general knowledge to reduce the demands for labeled data. It consists of four phases: 1) Model construction, where RENE builds LSTM-CRF model and an isomorphic LSTM language model for transfer learning; 2) LSTM language model training, where RENE captures general knowledge and adapt to requirement context; 3) LSTM-CRF training, where RENE trains the LSTM-CRF model with the transferred layers; 4) Requirement entity extraction, where RENE applies the trained LSTM-CRF model to a new-coming requirement, and automatically extracts its requirement entities. RENE is evaluated using two methods: evaluation on historical dataset and user study. The evaluation on the historical dataset shows that RENE could achieve 79% precision, 81% recall, and 80% F1. The evaluation results from the user study also suggest that RENE could produce more accurate and comprehensive requirement entities, compared with those produced by engineers.

The ability in rapidly learning and adapting to evolving user needs is key to modern business successes. Existing methods are based on text mining and machine learning techniques to analyze user comments and feedback, and often constrained by heavy reliance on manually codified rules or insufficient training data. Multitask learning (MTL) is an effective approach with many successful applications, with the potential to address these limitations associated with requirements analysis tasks. In this paper, we propose a deep MTL-based approach, DEMAR, to address these limitations when discovering requirements from massive issue reports and annotating the sentences in support of automated requirements analysis. DEMAR consists of three main phases: (1) data augmentation phase, for data preparation and allowing data sharing beyond single task learning; (2) model construction phase, for constructing the MTL-based model for requirements discovery and requirements annotation tasks; and (3) model training phase, enabling eavesdropping by shared loss function between the two related tasks. Evaluation results from eight open-source projects show that, the proposed multitask learning approach outperforms two state-of-the-art approaches (CNC and FRA) and six common machine learning algorithms, with the precision of 91% and the recall of 83% for requirements discovery task, and the overall accuracy of 83% for requirements annotation task. The proposed approach provides a novel and effective way to jointly learn two related requirements analysis tasks. We believe that it also sheds light on further directions of exploring multitask learning in solving other software engineering problems.

Requirements are usually written by different stakeholders with diverse backgrounds and skills and evolve continuously. Therefore inconsistency caused by specialized jargons and different domains, is inevitable. In particular, entity coreference in Requirement Engineering (RE) is that different linguistic expressions refer to the same real-world entity. It leads to misconception about technical terminologies, and impacts the readability and understandability of requirements negatively. Manual detection entity coreference is labor-intensive and time-consuming. In this paper, we propose a DEEP context-wise semantic method named DeepCoref to entity COREFerence detection. It consists of one fine-tuning BERT model for context representation and a Word2Vec-based network for entity representation. We use a multi-layer perception in the end to fuse and make a trade-off between two representations for obtaining a better representation of entities. The input of the network is requirement contextual text and related entities, and the output is the predictive label to infer whether two entities are coreferent. The evaluation on industry data shows that our approach significantly outperforms three baselines with average precision and recall of 96.10% and 96.06% respectively. We also compare DeepCoref with three variants to demonstrate the performance enhancement from different components.

In practice, it is very important to determine the size of a proposed software system yet to be built based on its requirements, i.e., early in the development life cycle. The most widely used approach for size estimation is Function Point Analysis (FPA). However, since FPA involves human judgment, the estimation results are some degree of subjective, and the process is labor and cost intensive. In this paper, we propose a novel approach to identify transaction functions from textual requirements automatically by leveraging a set of natural language processing techniques and machine learning models. We evaluate our approach on 1,864 requirements and 104,691 transaction functions taken from 36 financial projects from one banking industry. The results show that the contents of the suggested transaction functions by our approach are high in quality, with low perplexity value of 8.5 and high BLEU score of 34 on average. The types of suggested transaction functions can also be accurately classified, with overall accuracy of 0.99 on average. Our approach can provide reasonable suggestions that assist industrial practitioners to identify transaction functions faster and easier.

In open source communities, there are numerous duplicate issue reports, considered as useless and negligible by developers. Conversely, some researches argued that duplicates deliver complementary information that could benefit issue resolving. Considering all duplicates as value-neutral will result in either overestimation or underestimation of valuable information. It is necessary to be aware of whether all duplicates are redundant or beneficial. In this paper, we investigate whether duplicates have the same impacts on issue resolving and identification cost. We divide duplicates into three categories according to the statuses of master reports when duplicates are submitted. The results show duplicates in different categories play different roles in issue-resolving, and identification cost is also significantly different. Our study reveals duplicates are different, but almost are paid equal attentions. It is promising to propose new approaches and tools to resolve the problem.