We introduce a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning).Inspired by speech processing, these zero-shot tasks measure whether minimal differences in representation can be reliably detected.This offers a flexible and interpretable alternative to probing.Applied to XLM-R (Conneau et al., 2020a) across pretraining checkpoints and layers, we find that language discrimination declines over training and becomes concentrated in lower layers, while meaning discrimination strengthens over time and stabilizes in deeper layers.We then explore probing tasks, showing some alignment between our metrics and linguistic learning performance.Our results position ABX tasks as a lightweight framework for analyzing the structure of multilingual representations.

Current speech translation systems, while having achieved impressive accuracies, are rather static in their behavior and do not adapt to realworld situations in ways human interpreters do.In order to improve their practical usefulness and enable interpreting-like experiences, a precise understanding of the nature of human interpreting is crucial.To this end, we discuss human interpreting literature from the perspective of the machine translation field, while considering both operational and qualitative aspects.We identify implications for the development of speech translation systems and argue that there is great potential to adopt many human interpreting principles using recent modeling techniques.We hope that our findings provide inspiration for closing the perceived usability gap, and can motivate progress toward true machine interpreting.

We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis.We apply this to two tasks: speech resynthesis and speech-to-speech translation.In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language.As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.

ISSN: 2958-1796

International audience

We present the Zero Resource Speech Challenge 2021, which asks participants to learn a language model directly from audio, without any text or labels. The challenge is based on the Libri-light dataset, which provides up to 60k hours of audio from English audio books without any associated text. We provide a pipeline baseline system consisting on an encoder based on contrastive predictive coding (CPC), a quantizer ($k$-means) and a standard language model (BERT or LSTM). The metrics evaluate the learned representations at the acoustic (ABX discrimination), lexical (spot-the-word), syntactic (acceptability judgment) and semantic levels (similarity judgment). We present an overview of the eight submitted systems from four groups and discuss the main results.