Sina Ahmadi, Micha David Hess, Elena Álvarez-Mellado, Alessia Battisti, Cui Ding, Anne Göhring, Yingqiang Gao, Zifan Jiang, Andrianos Michail, Peshmerge Morad, Joel Niklaus, Maria Christina Panagiotopoulou, Stefano Perrella, Juri Opitz, Anastassia Shaitarova, Rico Sennrich. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

The task of determining whether two texts are paraphrases has long been a challenge in NLP. However, the prevailing notion of paraphrase is often quite simplistic, offering only a limited view of the vast spectrum of paraphrase phenomena. Indeed, we find that evaluating models in a paraphrase dataset can leave uncertainty about their true semantic understanding. To alleviate this, we create PARAPHRASUS, a benchmark designed for multi-dimensional assessment, benchmarking and selection of paraphrase detection models. We find that paraphrase detection models under our fine-grained evaluation lens exhibit trade-offs that cannot be captured through a single classification dataset. Furthermore, PARAPHRASUS allows prompt calibration for different use cases, tailoring LLM models to specific strictness levels. PARAPHRASUS includes 3 challenges spanning over 10 datasets, including 8 repurposed and 2 newly annotated; we release it along with a benchmarking library at https://github.com/impresso/paraphrasus

Controllable and transparent text generation has been a long-standing goal in NLP.Almost as long-standing is a general idea for addressing this challenge: Parsing text to a symbolic representation, and generating from it.However, earlier approaches were hindered by parsing and generation insufficiencies.Using modern parsers and a safety supervision mechanism, we show how close current methods come to this goal.Concretely, we propose the SENTENCE-SMITH framework for English, which has three steps: 1. Parsing a sentence into a semantic graph.2. Applying human-designed semantic manipulation rules.3. Generating text from the manipulated graph.A final entailment check (4.) verifies the validity of the applied transformation.To demonstrate our framework's utility, we use it to induce hard negative text pairs that challenge text embedding models.Since the controllable generation makes it possible to clearly isolate different types of semantic shifts, we can evaluate text embedding models in a fine-grained way, also addressing an issue in current benchmarking where linguistic phenomena remain opaque.Human validation confirms that our transparent generation process produces texts of good quality.Notably, our way of generation is very resource-efficient, since it relies only on smaller neural networks.

Text embeddings are a fundamental component in many NLP tasks, including classification, regression, clustering, and semantic search.However, despite their ubiquitous application, challenges persist in interpreting embeddings and explaining similarities between them.In this work, we provide a structured overview of methods specializing in inherently interpretable text embeddings and text similarity explanation, an underexplored research area.We characterize the main ideas, approaches, and tradeoffs.We compare means of evaluation, discuss overarching lessons learned and finally identify opportunities and open challenges for future research.