Machine translation quality estimation (QE) typically relies on dedicated neural models trained on human judgments. We evaluate whether cosine similarity over general-purpose embeddings can serve as a lightweight alternative, using Gemini embeddings as the scoring backbone. Through three experiments (rogue dimension analysis, score calibration, and a learned calibration head) and a root cause analysis, we find that cosine similarity between source and translation saturates in the 0.94–0.99 range because even poor translations preserve most of the source semantics, leaving an Area Under the ROC Curve (AUC) ceiling of approximately 0.63. However, a LightGBM classifier trained on normalized cosine and surface-level text features breaks through this ceiling (AUC 0.751), with the improvement driven primarily by features orthogonal to embedding similarity.

Diego Velazquez, Mikaela Grace, Konstantinos Karageorgos, Lawrence Carin, Aaron Schliem, Dimitrios Zaikis, Roger Wechsler. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Drug-Drug Interactions (DDI) identification is a part of the drug safety process, that focuses at avoiding potential adverse drug effects that can lead to patient health risks. With the exponential growth in published literature, it becomes increasingly difficult to extract useful information from the most relevant research. Therefore, Machine Learning and specifically Relationship Extraction has been employed for the extraction of DDIs from biomedical literature. This task consists of both Named Entity Recognition and Relationship Classification techniques tackled in either pipelined or joint approaches for recognizing drug mentions and classifying their interactions, respectively. However, current approaches are prone to error propagation between the tasks, not taking the relevance between them into account. In this paper we propose TransformDDI, an end-to-end Transformer-based joint multi-task DDI extraction model that integrates domain knowledge and a shared parameter layer in a dynamic drug entities extraction and interaction classification Language Model architecture. Our proposed model can generate variable outputs based on the recognized drug entities in a single-model architecture by implementing a Dynamic Pair Attention Mechanism with task-specific focus and dynamic loss functions. Experiments conducted on the DDI Extraction 2013 benchmark corpus indicate that our methodology offers significant improvements over the current state-of-the-art.