Yimin Xiao, Yongle Zhang, Dayeon Ki, Calvin Bao, Marianna J. Martindale, Charlotte Vaughn, Ge Gao, Marine Carpuat. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Recent advances in Machine Translation (MT) quality may motivate adoption in a variety of use cases, but the success of MT deployment depends not only on intrinsic model quality but on how well the model, as deployed, helps users meet the objectives of their use case. This work focuses on a specific triage use case, MT-enabled scanning in intelligence analysis. After describing the use case with its objectives and failure modes, we present a user study to establish a baseline performance level and measure the mitigating effects of a simple intervention, providing additional MT outputs. We find significant improvements in relevance judgment accuracy with outputs from two distinct neural MT models and significant improvements in relevant entity identification with the addition of a rule-based MT. Users also like seeing multiple MT outputs, making it an appealing way to improve MT-enabled scanning performance.

Abstract Neural sequence generation models are known to “hallucinate”, by producing outputs that are unrelated to the source text. These hallucinations are potentially harmful, yet it remains unclear in what conditions they arise and how to mitigate their impact. In this work, we first identify internal model symptoms of hallucinations by analyzing the relative token contributions to the generation in contrastive hallucinated vs. non-hallucinated outputs generated via source perturbations. We then show that these symptoms are reliable indicators of natural hallucinations, by using them to design a lightweight hallucination detector which outperforms both model-free baselines and strong classifiers based on quality estimation or large pre-trained models on manually annotated English-Chinese and German-English translation test beds.

A growing body of knowledge about biological mechanisms and interaction of biological components is contained in the peer-reviewed scientific literature. In order to leverage this knowledge towards the development of predictive models, one must first extract these relationships from the text. However, the context in which the interaction was reported is critical in ensuring that it is used in a manner consistent with the model's intended application. Here we assess the applicability of two generic automated methods for leveraging a broader contextual structure in the more specific domain of a biological experiment using only the paper's title and abstract. In an example use case, a Support Vector Machine (SVM) and two variants of the broadly-used Bidirectional Encoder Representations from Transformers (BERT) neural network model, serve to distinguish mouse from human subject experiments in a corpus of over 12,000 papers documenting mechanistic interactions in a regulatory model of of mucosal immune signaling. The BERT and domain-specific BioBERT yielded essentially equivalent classification accuracy with both neural network models performing only marginally better than the SVM. Words occurring frequently in abstracts were largely non-specific, whereas words unique to each class were used in 4% or less of the abstracts. These high-specificity words were used in very similar contexts that separated mouse and human study abstracts on the basis of study design and experimental procedure rather than species or basic biological markers.

Stylistic variations of language, such as formality, carry speakers' intention beyond literal meaning and should be conveyed adequately in translation.We propose to use lexical formality models to control the formality level of machine translation output.We demonstrate the effectiveness of our approach in empirical evaluations, as measured by automatic metrics and human assessments.