João Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Olympiad corpus. To attain high accuracy on this benchmark, models don't need previous knowledge of the tested language, as all the information needed to solve the linguistic puzzle is presented in the context. We find that, while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models, with the best-performing proprietary model scoring 24.05% and the best-performing open model 8.84%.

Speech-to-Speech and Speech-to-Text translation are currently dynamic areas of research. In our commitment to advance these fields, we present SpeechAlign, a framework designed to evaluate the underexplored field of source-target alignment in speech models. The SpeechAlign framework has two core components. First, to tackle the absence of suitable evaluation datasets, we introduce the Speech Gold Alignment dataset, built upon a English-German text translation gold alignment dataset. Secondly, we introduce two novel metrics, Speech Alignment Error Rate (SAER) and Time-weighted Speech Alignment Error Rate (TW-SAER), which enable the evaluation of alignment quality within speech models. While the former gives equal importance to each word, the latter assigns weights based on the length of the words in the speech signal. By publishing SpeechAlign we provide an accessible evaluation framework for model assessment, and we employ it to benchmark open-source Speech Translation models. In doing so, we contribute to the ongoing research progress within the fields of Speech-to-Speech and Speech-to-Text translation.

Direct speech-to-text translation systems encounter an important drawback in data scarcity.A common solution consists on pretraining the encoder on automatic speech recognition, hence losing efficiency in the training process.In this study, we compare the training dynamics of a system using a pretrained encoder, the conventional approach, and one trained from scratch.We observe that, throughout the training, the randomly initialized model struggles to incorporate information from the speech inputs for its predictions.Hence, we hypothesize that this issue stems from the difficulty of effectively training an encoder for direct speech translation.While a model trained from scratch needs to learn acoustic and semantic modeling simultaneously, a pretrained one can just focus on the latter.Based on these findings, we propose a subtle change in the decoder crossattention to integrate source information from earlier steps in training.We show that with this change, the model trained from scratch can achieve comparable performance to the pretrained one, while reducing the training time.

In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step).However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions.Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions.In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts.Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models.We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour.