Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Yi Sun, Luke Zettlemoyer, Gargi Ghosh, Wen-tau Yih. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Mutation analysis is a fundamental technique in software engineering, playing a vital role in software testing and debugging. It introduces small artificial faults (mutants) into a program to simulate realistic defects. These mutants are systematically injected and then leveraged in downstream tasks such as mutation testing, mutation-based fault localization (MBFL), and mutation-based test case prioritization (MBTCP). However, all of these applications require generating and executing a large number of mutants, leading to substantial computational overhead. This scalability issue significantly limits the applicability of mutation analysis to large-scale software systems. To address this challenge, we propose CAMUS, a context-aware neural mutation selection approach. CAMUS models each mutation and its surrounding context as a graph enriched with AST hierarchy and type information. It then employs an encoder with self-attention mechanisms to produce embeddings, from which the selection probability of each mutant is predicted. We evaluate CAMUS on three downstream tasks, i.e., mutation testing, MBFL, and MBTCP, by comparing it with state-of-the-art mutation selection techniques, including the recent large language model DeepSeek-v3. Experiments are conducted on the Defects4J 2.0 and ConDefects benchmarks. Our results show that CAMUS consistently outperforms existing methods across a wide range of mutation retention rates. In mutation testing, CAMUS achieves over 95% accuracy using only 50% of the mutants. In MBTCP, it reaches near-optimal prioritization performance with as little as 5% of the mutants, demonstrating its strong efficiency and effectiveness.

ABSTRACT Fault localization (FL) aims to automatically find the location of bugs in a software program. In the family of FL approaches, spectrum‐based fault localization (SBFL) is the most widely used and has been extensively studied, which computes the suspicious scores of a code element to be buggy via test coverage information. Mutation‐based fault localization (MBFL) further analyses the relationship between mutants and test results. Although MBFL involves more information, it also faces the following limitations. (1) To precisely evaluate suspicious sources, SBFL approaches proposed dozens of formulae based on coverage, while only a few of them have been adopted by MBFL. (2) The current MBFL approaches are based on the assumption that the capability in localizing bugs of each mutant is the same, so they assign the same weight to the mutants that change the test outputs and consider the mutant from the same code element equivalent. In this paper, we intend to enrich the MBFL family by the following two approaches. First, we collect 25 typical SBFL formulae and transform them into MBFL versions. Second, we propose new assumptions that the mutants should have different weights in computing suspicious sources and propose BLMu, a novel MBFL approach that treats mutants differently. We also propose novel metrics for MBFL by considering the high cost of mutation analysis. We perform large‐scale experiments by evaluating all the MBFL approaches against 395 real‐world Java bugs of the Defects4J benchmark. Our evaluation results reveal that BLMu demonstrates a substantial improvement over both MUSE and Metallaxis, the most popular MBFL approaches, at both the statement level and the method level. Specifically, in terms of Top‐1 , BLMu improves by 106% at the statement level and 69% at the method level. When compared with other types of FL approaches, MBFL outperforms typical SBFL approaches while still far behind the state‐of‐the‐art learning‐based FL approaches.

Few-shot learning for open domain multi-hop question answering typically relies on the incontext learning capability of large language models (LLMs).While powerful, these LLMs usually contain tens or hundreds of billions of parameters, making them rather inefficient at inference time.To improve performance of smaller language models, we propose a data synthesis framework for multi-hop question answering that requires less than 10 humanannotated question answer pairs.Our framework depends only on rich, naturally-occurring relationships among documents and is built upon the data generation functions parameterized by LLMs and prompts.We synthesize millions of multi-hop questions and claims to finetune language models, evaluated on popular benchmarks for multi-hop question answering and fact verification.Empirically, our approach improves model performance significantly, allowing the finetuned models to be competitive with GPT-3.5 based approaches while being almost one-third the size in parameter count.What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?Query1: the eastern section of the Colorado orogeny Query2: the elevation range for the High Plains Unanswerable Question generation Query generation Query verification Question: What is the elevation range for … Query: the eastern section of the Colorado … Observation: 1: The Colorado orogeny, or Colorado orogen, was an orogeny … … Answer: 1,800 to 7,000 ft Question answering 1,800 to 7,000 ft Randomly sample document pairsThe causes of World War II are debated, but contributing factors included the Second Italo-Ethiopian War, Spanish Civil War, Second Sino-Japanese War, Soviet-Japanese border conflicts, the rise of fascism in Europe, and European tensions in the aftermath of World War I.The Second Italo-Ethiopian War, also referred to as the Second Italo-Abyssinian War, was a war of aggression which was fought between Italy and Ethiopia from October 1935 to February 1937. Events occurred in sequenceThe Colorado orogeny, or Colorado orogen, was an orogeny in Colorado and surrounding areas which was a part of the development of the ancestral Rockies.The eastern sector extends into the High Plains and is called the Central Plains orogeny.The High Plains are a subregion of the Great Plains.From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m). Extra geographical informationNew York, often called New York City or NYC, is the most populous city in the United States.With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km 2 ), the city is the most densely populated major city in the United States.NYC is more than twice as populous as Los Angeles, the nation's second-largest city.Los Angeles, often referred to by its initials L.A., officially the City of Los Angeles, is the most populous city in the U.S. state of California. Extra demographic information

Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk, Marta R. Costa-jussà. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

We introduce SUMMSCREEN, a summarization dataset comprised of pairs of TV series transcripts and human written recaps.The dataset provides a challenging testbed for abstractive summarization for several reasons.Plot details are often expressed indirectly in character dialogues and may be scattered across the entirety of the transcript.These details must be found and integrated to form the succinct plot descriptions in the recaps.Also, TV scripts contain content that does not directly pertain to the central plot but rather serves to develop characters or provide comic relief.This information is rarely contained in recaps.Since characters are fundamental to TV series, we also propose two entity-centric evaluation metrics.Empirically, we characterize the dataset by evaluating several methods, including neural models and those based on nearest neighbors.An oracle extractive approach outperforms all benchmarked models according to automatic metrics, showing that the neural models are unable to fully exploit the input transcripts.Human evaluation and qualitative analysis reveal that our nonoracle models are competitive with their oracle counterparts in terms of generating faithful plot events and can benefit from better content selectors.Both oracle and non-oracle models generate unfaithful facts, suggesting future research directions.

Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

We present a large-scale dataset for the task of rewriting an ill-formed natural language question to a well-formed one. Our multi-domain question rewriting (MQR) dataset is constructed from human contributed Stack Exchange question edit histories. The dataset contains 427,719 question pairs which come from 303 domains. We provide human annotations for a subset of the dataset as a quality estimate. When moving from ill-formed to well-formed questions, the question quality improves by an average of 45 points across three aspects. We train sequence-to-sequence neural models on the constructed dataset and obtain an improvement of 13.2% in BLEU-4 over baseline methods built from other data resources. We release the MQR dataset to encourage research on the problem of question rewriting.1

Prior work on controllable text generation usually assumes that the controlled attribute can take on one of a small set of values known a priori.In this work, we propose a novel task, where the syntax of a generated sentence is controlled rather by a sentential exemplar.To evaluate quantitatively with standard metrics, we create a novel dataset with human annotations.We also develop a variational model with a neural module specifically designed for capturing syntactic knowledge and several multitask training objectives to promote disentangled representation learning.Empirically, the proposed model is observed to achieve improvements over baselines and learn to capture desirable characteristics.

Mingda Chen, Zewei Chu, Yang Chen, Karl Stratos, Kevin Gimpel. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Mingda Chen, Zewei Chu, Kevin Gimpel. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We introduce a family of multitask variational methods for semi-supervised sequence labeling.Our model family consists of a latentvariable generative model and a discriminative labeler.The generative models use latent variables to define the conditional probability of a word given its context, drawing inspiration from word prediction objectives commonly used in learning word embeddings.The labeler helps inject discriminative information into the latent space.We explore several latent variable configurations, including ones with hierarchical structure, which enables the model to account for both label-specific and word-specific information.Our models consistently outperform standard sequential baselines on 8 sequence labeling datasets, and improve further with unlabeled data.