We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language.We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts.We evaluate a range of LLMs on monolingual and crosslingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language.We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures.We find that language confusion can be partially mitigated via fewshot prompting, multilingual SFT and preference tuning.We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation. 1

The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages.The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs.It contains 54.5B parameters and requires at least four 32GB GPUs just for inference.In this work, we propose a pruning method that enables the removal of up to 80% of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU.Further analysis suggests that our pruning metrics can identify language-specific experts.

CompCert (ACM Software System Award 2021) is the first industrial-strength compiler with a mechanically checked proof of correctness. Yet, CompCert remains a moderately optimizing C compiler. Indeed, some optimizations of “gcc ‍-O1” such as Lazy Code Motion (LCM) or Strength Reduction (SR) were still missing: developing these efficient optimizations together with their formal proofs remained a challenge. Cyril Six et al. have developed efficient formally verified translation validators for certifying the results of superblock schedulers and peephole optimizations. We revisit and generalize their approach into a framework (integrated into CompCert) able to validate many more optimizations: an enhanced superblock scheduler, but also Dead Code Elimination (DCE), Constant Propagation (CP), and more noticeably, LCM and SR. In contrast to other approaches to translation validation, we co-design our untrusted optimizations and their validators. Our optimizations provide hints, in the forms of invariants or CFG morphisms , that help keep the formally verified validators both simple and efficient. Such designs seem applicable beyond CompCert.

Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, Laurent Besacier. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Multilingual NMT has become an attractive solution for MT deployment in production.But to match bilingual quality, it comes at the cost of larger and slower models.In this work, we consider several ways to make multilingual NMT faster at inference without degrading its quality.We experiment with several "light decoder" architectures in two 20language multi-parallel settings: small-scale on TED Talks and large-scale on ParaCrawl.Our experiments demonstrate that combining a shallow decoder with vocabulary filtering leads to more than ×2 faster inference with no loss in translation quality.We validate our findings with BLEU and chrF (on 380 language pairs), robustness evaluation and human evaluation.

We consider the problem of multilingual unsupervised machine translation, translating to and from languages that only have monolingual data by using auxiliary parallel language pairs.For this problem the standard procedure so far to leverage the monolingual data is back-translation, which is computationally costly and hard to tune.In this paper we propose instead to use denoising adapters, adapter layers with a denoising objective, on top of pre-trained mBART-50.In addition to the modularity and flexibility of such an approach we show that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally.

We propose a novel adapter layer formalism for adapting multilingual models.They are more parameter-efficient than existing adapter layers while obtaining as good or better performance.The layers are specific to one language (as opposed to bilingual adapters) allowing to compose them and generalize to unseen language-pairs.In this zero-shot setting, they obtain a median improvement of +2.77BLEU points over a strong 20-language multilingual Transformer baseline trained on TED talks.

We investigate end-to-end speech-to-text translation on a corpus of audiobooks specifically augmented for this task. Previous works investigated the extreme case where source language transcription is not available during learning nor decoding, but we also study a midway case where source language transcription is available at training time only. In this case, a single model is trained to decode source speech into target text in a single pass. Experimental results show that it is possible to train compact and efficient end-to-end speech translation models in this setup. We also distribute the corpus and hope that our speech translation baseline on this corpus will be challenged in the future.

International audience

Word discovery is the task of extracting words from un-segmented text. In this paper we examine to what extent neural networks can be applied to this task in a realistic unwritten language scenario, where only small corpora and limited annotations are available. We investigate two scenarios: one with no supervision and another with limited supervision with access to the most frequent words. Obtained results show that it is possible to retrieve at least 27% of the gold standard vocabulary by training an encoder-decoder neural machine translation system with only 5,157 sentences. This result is close to those obtained with a task-specific Bayesian nonparametric model. Moreover, our approach has the advantage of generating translation alignments, which could be used to create a bilingual lexicon. As a future perspective, this approach is also well suited to work directly from speech.

This paper presents an approach combining lexico-semantic resources and distributed representations of words applied to the evaluation in machine translation (MT). This study is made through the enrichment of a well-known MT evaluation metric: METEOR. METEOR enables an approximate match (synonymy or morphological similarity) between an automatic and a reference translation. Our experiments are made in the framework of the Metrics task of WMT 2014. We show that distributed representations are a good alternative to lexico-semanticresources for MT evaluation and they can even bring interesting additional information. The augmented versions of METEOR, using vector representations, are made available on our Github page.