In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2019) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language.To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs.From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Most machine translation systems construct translations from a closed vocabulary of target word forms, posing problems for translating into languages that have productive compounding processes.We present a simple and effective approach that deals with this problem in two phases.First, we build a classifier that identifies spans of the input text that can be translated into a single compound word in the target language.Then, for each identified span, we generate a pool of possible compounds which are added to the translation model as "synthetic" phrase translations.Experiments reveal that (i) we can effectively predict what spans can be compounded; (ii) our compound generation model produces good compounds; and (iii) modest improvements are possible in end-to-end English-German and English-Finnish translation tasks.We additionally introduce KomposEval, a new multi-reference dataset of English phrases and their translations into German compounds.

Translation into morphologically rich languages is an important but recalcitrant problem in MT.We present a simple and effective approach that deals with the problem in two phases.First, a discriminative model is learned to predict inflections of target words from rich source-side annotations.Then, this model is used to create additional sentencespecific word-and phrase-level translations that are added to a standard translation model as "synthetic" phrases.Our approach relies on morphological analysis of the target language, but we show that an unsupervised Bayesian model of morphology can successfully be used in place of a supervised analyzer.We report significant improvements in translation quality when translating from English to Russian, Hebrew and Swahili.